Nov 03 13:03:42.610691 ip-10-0-129-63 systemd[1]: Starting Kubernetes Kubelet... Nov 03 13:03:43.002477 ip-10-0-129-63 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:03:43.002477 ip-10-0-129-63 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 13:03:43.002477 ip-10-0-129-63 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:03:43.002477 ip-10-0-129-63 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:03:43.005247 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005163 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:43.005247 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005247 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005252 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005256 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005259 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005262 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005266 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005269 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005271 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005274 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005277 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005281 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005285 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005288 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005291 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005294 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005297 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005299 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005302 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005305 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005307 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:43.005319 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005310 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005313 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005315 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005318 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005321 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005324 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005327 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005330 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005333 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005335 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005346 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005349 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005352 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005354 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005357 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005360 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005362 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005365 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005368 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:43.005802 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005370 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005373 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005375 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005378 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005380 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005397 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005400 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005402 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005405 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005407 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005410 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005414 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005416 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005419 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005422 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005427 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005432 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005436 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005439 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:43.006241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005442 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005445 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005447 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005450 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005453 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005456 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005459 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005462 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005464 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005467 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005469 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005472 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005474 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005477 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005479 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005482 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005485 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005487 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005489 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005492 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:43.006696 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005495 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005497 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005500 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005503 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005506 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005508 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005511 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005513 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005517 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005520 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005522 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005525 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005527 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005530 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005532 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005535 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005538 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005540 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005543 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:43.007167 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005545 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005548 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005550 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005552 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005555 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005557 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005560 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.005562 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006403 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006411 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006414 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006418 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006420 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006423 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006426 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006429 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006432 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006435 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006438 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006440 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:43.007616 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006443 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006446 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006449 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006452 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006455 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006458 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006460 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006463 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006466 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006469 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006471 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006474 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006476 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006479 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006481 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006484 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006486 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006490 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006492 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:43.008096 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006495 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006498 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006500 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006503 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006505 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006508 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006511 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006514 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006516 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006519 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006522 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006525 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006528 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006530 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006533 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006536 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006538 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006541 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006543 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:43.008570 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006546 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006548 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006551 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006554 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006556 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006559 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006561 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006564 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006567 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006570 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006572 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006575 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006577 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006580 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006582 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006585 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006587 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006592 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006595 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006598 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:43.009014 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006601 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006603 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006606 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006608 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006612 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006614 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006617 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006620 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006623 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006625 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006628 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006631 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006633 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006635 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006638 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006640 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006643 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006646 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006648 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:43.009497 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006651 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006653 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006656 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006658 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006661 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006663 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006665 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006668 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006670 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006673 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006677 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006681 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006684 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006687 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006690 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006692 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.006694 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007187 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007192 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:43.009957 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007195 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007198 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007201 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007203 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007206 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007209 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007212 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007215 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007217 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007220 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007223 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007225 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007228 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007230 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007233 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007236 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007238 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007241 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007243 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:43.010419 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007246 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007248 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007251 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007253 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007256 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007258 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007261 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007263 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007266 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007268 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007271 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007274 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007277 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007279 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007285 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007287 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007290 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007293 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007295 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:43.010866 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007298 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007300 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007303 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007306 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007309 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007311 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007314 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007316 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007319 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007322 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007324 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007327 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007329 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007332 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007334 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007337 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007339 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007342 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007344 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007347 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:43.011349 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007349 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007352 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007354 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007357 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007359 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007362 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007368 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007373 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007376 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007378 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007394 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007397 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007399 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007402 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007405 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007408 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007411 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007413 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007416 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:43.011834 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007420 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007424 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007427 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007430 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007432 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007435 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007438 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007441 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007445 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007449 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007452 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007454 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007457 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007459 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007462 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007467 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007469 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007472 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007475 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:43.012286 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007477 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007483 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007485 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007490 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007492 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007495 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007498 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.007500 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007589 2582 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007597 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007603 2582 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007608 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007613 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007616 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007625 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007629 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007632 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007635 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007639 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007643 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007646 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007649 2582 flags.go:64] FLAG: --cgroup-root="" Nov 03 13:03:43.012785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007652 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007655 2582 flags.go:64] FLAG: --client-ca-file="" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007657 2582 flags.go:64] FLAG: --cloud-provider="external" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007660 2582 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007664 2582 flags.go:64] FLAG: --cluster-domain="" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007667 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007672 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007675 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007678 2582 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007682 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007685 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007689 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007693 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007696 2582 flags.go:64] FLAG: --contention-profiling="false" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007699 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007702 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007705 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007708 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007712 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007715 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007718 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007721 2582 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007724 2582 flags.go:64] FLAG: --enable-server="true" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007727 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007731 2582 flags.go:64] FLAG: --event-burst="100" Nov 03 13:03:43.013359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007735 2582 flags.go:64] FLAG: --event-qps="50" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007737 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007740 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007743 2582 flags.go:64] FLAG: --eviction-hard="" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007747 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007750 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007753 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007756 2582 flags.go:64] FLAG: --eviction-soft="" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007759 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007762 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007765 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007767 2582 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007770 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007775 2582 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007777 2582 flags.go:64] FLAG: --feature-gates="" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007782 2582 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007785 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007788 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007791 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007794 2582 flags.go:64] FLAG: --healthz-port="10248" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007804 2582 flags.go:64] FLAG: --help="false" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007807 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007810 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007813 2582 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 13:03:43.014075 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007816 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007820 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007824 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007827 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007829 2582 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007833 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007835 2582 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007839 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007842 2582 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007845 2582 flags.go:64] FLAG: --kube-reserved="" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007848 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007850 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007854 2582 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007857 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007859 2582 flags.go:64] FLAG: --lock-file="" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007863 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007866 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007869 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007874 2582 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007877 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007880 2582 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007885 2582 flags.go:64] FLAG: --logging-format="text" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007888 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007891 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 13:03:43.014651 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007893 2582 flags.go:64] FLAG: --manifest-url="" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007896 2582 flags.go:64] FLAG: --manifest-url-header="" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007901 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007904 2582 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007914 2582 flags.go:64] FLAG: --max-pods="110" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007924 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007927 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007930 2582 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007933 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007936 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007939 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007942 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007949 2582 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007952 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007955 2582 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007957 2582 flags.go:64] FLAG: --pod-cidr="" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007960 2582 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007964 2582 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007967 2582 flags.go:64] FLAG: --pods-per-core="0" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007969 2582 flags.go:64] FLAG: --port="10250" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007972 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007975 2582 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-037071a651035b0b9" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007978 2582 flags.go:64] FLAG: --qos-reserved="" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007981 2582 flags.go:64] FLAG: --read-only-port="10255" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007984 2582 flags.go:64] FLAG: --register-node="true" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007987 2582 flags.go:64] FLAG: --register-with-taints="" Nov 03 13:03:43.015224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007991 2582 flags.go:64] FLAG: --registry-burst="10" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007993 2582 flags.go:64] FLAG: --registry-qps="5" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.007996 2582 flags.go:64] FLAG: --reserved-cpus="" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008001 2582 flags.go:64] FLAG: --reserved-memory="" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008004 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008007 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008011 2582 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008014 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008016 2582 flags.go:64] FLAG: --runonce="false" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008019 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008022 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008025 2582 flags.go:64] FLAG: --seccomp-default="false" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008034 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008037 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008041 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008044 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008047 2582 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008050 2582 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008053 2582 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008055 2582 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008058 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008061 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008064 2582 flags.go:64] FLAG: --system-cgroups="" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008067 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008072 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 13:03:43.015855 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008075 2582 flags.go:64] FLAG: --tls-cert-file="" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008077 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008084 2582 flags.go:64] FLAG: --tls-min-version="" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008087 2582 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008089 2582 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008092 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008095 2582 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008098 2582 flags.go:64] FLAG: --v="2" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008104 2582 flags.go:64] FLAG: --version="false" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008108 2582 flags.go:64] FLAG: --vmodule="" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008114 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008117 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008298 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008302 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008306 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008309 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008311 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008314 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008316 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008319 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008327 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008330 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008333 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:43.016445 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008335 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008338 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008340 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008343 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008346 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008348 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008351 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008353 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008356 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008358 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008361 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008364 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008368 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008372 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008395 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008398 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008401 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008405 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008408 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:43.016995 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008412 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008415 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008417 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008420 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008423 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008425 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008428 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008430 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008433 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008436 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008438 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008447 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008450 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008452 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008455 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008458 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008460 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008463 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008467 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:43.017492 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008472 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008475 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008478 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008480 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008483 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008486 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008488 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008491 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008493 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008496 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008499 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008503 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008506 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008509 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008512 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008515 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008517 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008520 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008523 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008525 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:43.017951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008528 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008531 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008533 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008536 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008538 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008547 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008549 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008552 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008555 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008558 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008560 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008563 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008565 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008568 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008571 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008573 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008576 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008579 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008582 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008584 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:43.018424 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008587 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008589 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008592 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008596 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008599 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008602 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008605 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008607 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008610 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008613 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008615 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008618 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008620 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008623 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008625 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008628 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:43.018880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008630 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.008657 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008713 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008718 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008722 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008724 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008727 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008730 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008732 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008735 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008738 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008741 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008743 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008746 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008749 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008751 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:43.019296 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008754 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008756 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008759 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008761 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008765 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008768 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008771 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008774 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008776 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008779 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008781 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008784 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008786 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008789 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008791 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008794 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008797 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008799 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008802 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008810 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:43.019686 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008813 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008816 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008818 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008821 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008823 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008826 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008828 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008831 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008833 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008836 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008839 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008841 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008844 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008846 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008848 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008851 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008855 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008857 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008860 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008864 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:43.020154 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008868 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008871 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008873 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008876 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008879 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008881 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008884 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008886 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008889 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008892 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008894 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008897 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008905 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008908 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008910 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008913 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008916 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008918 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008921 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:43.020711 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008923 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008926 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008930 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008932 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008935 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008939 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008942 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008945 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008948 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008952 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008954 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008957 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008959 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008962 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008965 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008967 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008970 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008973 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008975 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:43.021201 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008978 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008980 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008983 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008985 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008988 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008990 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.008993 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.009001 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.009004 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.009009 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.009012 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.009018 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.009021 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.009024 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.015711 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.015726 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.015740 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.015746 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015806 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015811 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:43.021812 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015815 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015818 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015821 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015824 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015826 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015829 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015832 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015835 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015837 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015840 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015843 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015846 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015848 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015851 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015854 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015858 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015862 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015866 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015869 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:43.022275 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015873 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015876 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015879 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015881 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015884 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015887 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015890 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015893 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015896 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015899 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015902 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015920 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015924 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015927 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015930 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015933 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015935 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015938 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015941 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015944 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:43.022757 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015946 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015949 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015952 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015955 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015957 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015960 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015963 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015965 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015968 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015970 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015973 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015976 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015978 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015981 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015985 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015989 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015992 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015995 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.015998 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016001 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:43.023259 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016004 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016008 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016011 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016014 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016017 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016019 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016022 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016024 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016027 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016029 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016032 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016035 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016038 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016040 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016043 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016045 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016048 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016050 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016052 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016055 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:43.023981 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016057 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016060 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016062 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016065 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016067 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016070 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016072 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016075 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016078 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016080 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016083 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016085 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016088 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016091 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016093 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016096 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016099 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016101 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016104 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:43.024880 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016106 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016109 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016111 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016114 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016116 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016119 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.016146 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016287 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016292 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016295 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016298 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016301 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016304 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016306 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016309 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016311 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:43.025441 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016314 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016317 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016319 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016322 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016325 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016327 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016330 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016333 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016335 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016338 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016341 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016343 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016346 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016349 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016352 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016354 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016357 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016359 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016362 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:43.025867 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016365 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016367 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016370 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016372 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016375 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016377 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016393 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016396 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016400 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016404 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016407 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016410 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016413 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016415 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016418 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016420 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016423 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016426 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016429 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:43.026425 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016432 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016434 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016437 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016440 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016442 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016445 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016448 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016451 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016453 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016456 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016459 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016462 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016465 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016468 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016471 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016473 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016476 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016478 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016481 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016483 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:43.026890 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016486 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016489 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016491 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016494 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016497 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016500 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016502 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016505 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016507 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016510 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016512 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016515 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016518 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016520 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016523 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016525 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016528 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016531 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016533 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016536 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:43.027414 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016538 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016541 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016543 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016546 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016548 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016551 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016554 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016556 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016559 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016561 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016564 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016566 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016569 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016571 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016575 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016579 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016582 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016585 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:43.027980 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016588 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.016611 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016644 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016649 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016653 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016656 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016659 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016662 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016665 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016668 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016675 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016678 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016680 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016683 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016686 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016689 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:43.028734 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016692 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016694 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016697 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016700 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016703 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016705 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016708 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016710 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016713 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016716 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016719 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016721 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016724 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016727 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016729 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016732 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016734 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016737 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016739 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:43.029123 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016742 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016746 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016749 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016752 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016754 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016758 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016760 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016765 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016768 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016770 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016773 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016776 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016779 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016781 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016784 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016786 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016789 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016792 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016794 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:43.029591 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016797 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016799 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016802 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016805 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016808 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016811 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016813 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016816 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016819 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016822 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016824 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016827 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016829 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016832 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016835 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016837 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016840 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016843 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016845 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016849 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:43.030088 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016853 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016855 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016858 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016860 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016863 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016866 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016868 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016871 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016873 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016876 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016879 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016881 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016883 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016886 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016889 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016891 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016894 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016896 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016899 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016903 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:43.030623 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016906 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016908 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016911 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016913 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016916 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016918 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016921 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016926 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016929 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016932 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016936 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016938 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016943 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:43.016946 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.016996 2582 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.018992 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.020217 2582 server.go:1014] "Starting client certificate rotation" Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.020337 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:03:43.031110 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.021161 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:03:43.031558 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.022069 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:03:43.031558 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.025206 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:03:43.034523 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.034503 2582 log.go:25] "Validated CRI v1 runtime API" Nov 03 13:03:43.037847 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.037832 2582 log.go:25] "Validated CRI v1 image API" Nov 03 13:03:43.039871 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.039840 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 13:03:43.044365 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.044344 2582 fs.go:135] Filesystem UUIDs: map[1068437d-1ad0-47b3-b577-4f35c644470f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 987425b1-716e-4549-9735-e8109ec8d371:/dev/nvme0n1p4] Nov 03 13:03:43.044435 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.044365 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 13:03:43.050447 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.050325 2582 manager.go:217] Machine: {Timestamp:2026-11-03 13:03:43.048505804 +0000 UTC m=+0.322630926 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3078212 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22aba239f9628c35cec9706a235fb5 SystemUUID:ec22aba2-39f9-628c-35ce-c9706a235fb5 BootID:3a59fc5f-7f52-487c-b830-5dcf1f9b0cb5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c3:ea:31:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c3:ea:31:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:05:94:8f:55:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 13:03:43.050447 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.050446 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 13:03:43.050568 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.050556 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 13:03:43.054026 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.054005 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 13:03:43.054194 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.054028 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-63.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 13:03:43.054234 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.054197 2582 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 13:03:43.054234 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.054205 2582 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 13:03:43.054286 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.054234 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:03:43.054286 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.054253 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:03:43.054952 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.054942 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 13:03:43.059019 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.059008 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 13:03:43.059157 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.059148 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 13:03:43.059909 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.059898 2582 kubelet.go:493] "Attempting to sync node with API server" Nov 03 13:03:43.059948 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.059915 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 13:03:43.059948 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.059926 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 13:03:43.059948 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.059936 2582 kubelet.go:405] "Adding apiserver pod source" Nov 03 13:03:43.059948 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.059944 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 13:03:43.060277 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.060259 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:03:43.062506 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.062489 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 13:03:43.067019 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.067004 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:03:43.067777 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.067764 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 13:03:43.067812 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.067807 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 13:03:43.067965 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.067954 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 13:03:43.068004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.067976 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 13:03:43.068004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.067986 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 13:03:43.068004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.067995 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 13:03:43.068078 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068012 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 13:03:43.068078 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068024 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 13:03:43.068416 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068406 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 13:03:43.068416 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068417 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 13:03:43.068478 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068424 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 13:03:43.068478 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068430 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 13:03:43.068478 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068444 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 13:03:43.068478 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068452 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 13:03:43.068958 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068950 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 13:03:43.068958 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.068958 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 13:03:43.072922 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.072907 2582 server.go:1263] "Started kubelet" Nov 03 13:03:43.073025 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.072989 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 13:03:43.073084 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.073002 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 13:03:43.075654 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.075488 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 13:03:43.076424 ip-10-0-129-63 systemd[1]: Started Kubernetes Kubelet. Nov 03 13:03:43.077614 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.077046 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 13:03:43.083279 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.083253 2582 server.go:317] "Adding debug handlers to kubelet server" Nov 03 13:03:43.083430 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.083409 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 13:03:43.083730 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.083714 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 13:03:43.083807 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.083738 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 13:03:43.083861 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.083838 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:43.084578 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084353 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 13:03:43.084578 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084367 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 13:03:43.084578 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084494 2582 reconstruct.go:97] "Volume reconstruction finished" Nov 03 13:03:43.084578 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084501 2582 reconciler.go:29] "Reconciler: start to sync state" Nov 03 13:03:43.084578 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084557 2582 factory.go:55] Registering systemd factory Nov 03 13:03:43.084578 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084572 2582 factory.go:223] Registration of the systemd container factory successfully Nov 03 13:03:43.084866 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084714 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 13:03:43.084866 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084788 2582 factory.go:153] Registering CRI-O factory Nov 03 13:03:43.084866 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084802 2582 factory.go:223] Registration of the crio container factory successfully Nov 03 13:03:43.084964 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084900 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 13:03:43.084964 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084921 2582 factory.go:103] Registering Raw factory Nov 03 13:03:43.084964 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.084936 2582 manager.go:1199] Started watching for new ooms in manager Nov 03 13:03:43.085662 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.085651 2582 manager.go:319] Starting recovery of all containers Nov 03 13:03:43.095164 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.095146 2582 manager.go:324] Recovery completed Nov 03 13:03:43.100542 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.099329 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:43.101439 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.101421 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:43.101519 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.101457 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:43.101519 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.101473 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:43.102299 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.102006 2582 cpu_manager.go:226] "Starting" policy="none" Nov 03 13:03:43.102299 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.102017 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 13:03:43.102299 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.102023 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:43.102299 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.102036 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 13:03:43.104287 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.104273 2582 policy_none.go:50] "Start" Nov 03 13:03:43.104342 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.104293 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 13:03:43.104342 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.104311 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 13:03:43.108031 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.108014 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:43.108092 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.108047 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:43.108092 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.108061 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.147742 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.148868 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.148897 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.148920 2582 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.148962 2582 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.184224 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.210253 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-63.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.210252 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-63.us-west-2.compute.internal.18e48aac07f62929 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-63.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-63.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:03:43.072880937 +0000 UTC m=+0.347006060,LastTimestamp:2026-11-03 13:03:43.072880937 +0000 UTC m=+0.347006060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-63.us-west-2.compute.internal,}" Nov 03 13:03:43.247843 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.246945 2582 policy_none.go:44] "Start" Nov 03 13:03:43.250829 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.249775 2582 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 13:03:43.256479 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.256455 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-63.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 13:03:43.259803 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.259779 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 13:03:43.259867 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.259852 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-63.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 13:03:43.260116 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.260051 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-63.us-west-2.compute.internal.18e48aac09a9ed56 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-63.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-63.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-63.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:03:43.101439318 +0000 UTC m=+0.375564448,LastTimestamp:2026-11-03 13:03:43.101439318 +0000 UTC m=+0.375564448,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-63.us-west-2.compute.internal,}" Nov 03 13:03:43.285169 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.285146 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:43.286624 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.286584 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 13:03:43.288811 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.288798 2582 manager.go:341] "Starting Device Plugin manager" Nov 03 13:03:43.288913 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.288895 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 13:03:43.288913 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.288911 2582 server.go:87] "Starting device plugin registration server" Nov 03 13:03:43.289191 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.289177 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 13:03:43.289259 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.289192 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 13:03:43.289332 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.289317 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 13:03:43.289424 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.289402 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 13:03:43.289424 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.289416 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 13:03:43.290024 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.290005 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 13:03:43.290106 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.290052 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:43.304078 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.304049 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 13:03:43.304185 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.304110 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wzj7c" Nov 03 13:03:43.375417 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.375358 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:43.389745 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.389727 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:43.390725 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.390708 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:43.390816 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.390740 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:43.390816 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.390751 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:43.390816 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.390774 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.450119 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.450095 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal"] Nov 03 13:03:43.450194 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.450160 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:43.451071 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.451056 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:43.451144 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.451085 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:43.451144 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.451094 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:43.453425 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.453413 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:43.453530 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.453515 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.453572 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.453556 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:43.454199 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.454178 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:43.454199 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.454187 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:43.454291 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.454210 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:43.454291 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.454212 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:43.454291 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.454219 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:43.454291 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.454225 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:43.456217 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.456204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.456270 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.456228 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:43.457672 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.457655 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:43.457722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.457687 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:43.457722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.457697 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:43.481571 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.481552 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:43.481977 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.481959 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.482021 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.481991 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-63.us-west-2.compute.internal\": node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:43.485665 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.485643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.485766 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.485674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df29524ca054c489efb3ae39305cb76-config\") pod \"kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"8df29524ca054c489efb3ae39305cb76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.485766 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.485729 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.486478 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.486456 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" node="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.489007 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.488989 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" node="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.586707 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.586622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.586707 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.586660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.586707 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.586685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df29524ca054c489efb3ae39305cb76-config\") pod \"kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"8df29524ca054c489efb3ae39305cb76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.586894 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.586738 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df29524ca054c489efb3ae39305cb76-config\") pod \"kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"8df29524ca054c489efb3ae39305cb76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.586894 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.586746 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.586894 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.586738 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.599809 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.599790 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:43.700467 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.700440 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:43.789775 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.789742 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.793198 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:43.793180 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:43.801559 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:43.801439 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:45.383272 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:45.383203 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:45.483655 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:45.483625 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:45.500781 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.500761 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 13:03:45.500902 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.500878 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 13:03:45.500937 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.500885 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 13:03:45.500980 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:45.500902 2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a16aeb6c627d247dd9f516a896748b1c-7174c5b4c9fb5a11.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-63.us-west-2.compute.internal.18e48aac09aa878d\": read tcp 10.0.129.63:36940->18.224.78.59:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-63.us-west-2.compute.internal.18e48aac09aa878d default 4810 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-63.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-63.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-63.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:03:43 +0000 UTC,LastTimestamp:2026-11-03 13:03:43.390754557 +0000 UTC m=+0.664879680,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-63.us-west-2.compute.internal,}" Nov 03 13:03:45.584707 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:45.584673 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:45.685142 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:45.685117 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:45.769140 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.769105 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:45.770479 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.770451 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:45.770599 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.770495 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:45.770599 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.770508 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:45.786191 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:45.786170 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:45.789330 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.789307 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 12:58:44 +0000 UTC" deadline="2028-04-25 20:51:46.712668127 +0000 UTC" Nov 03 13:03:45.789377 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:45.789332 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12943h48m0.923339521s" Nov 03 13:03:45.886414 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:45.886389 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:45.986792 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:45.986723 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:46.087403 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.087378 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:46.188032 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.188004 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:03:46.222272 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.222240 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:46.265392 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.265335 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:46.327292 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.327266 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:03:46.328045 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.328030 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:03:46.339282 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.339266 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:46.386029 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.386002 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:03:46.476940 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:46.476906 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d4fd5bce6fe0026d1ea2d7ee6f2f03.slice/crio-cc857669bc04ba5000b5a232dc118f91b3248463f19cff875d13c7711cc6844d WatchSource:0}: Error finding container cc857669bc04ba5000b5a232dc118f91b3248463f19cff875d13c7711cc6844d: Status 404 returned error can't find the container with id cc857669bc04ba5000b5a232dc118f91b3248463f19cff875d13c7711cc6844d Nov 03 13:03:46.477306 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:46.477276 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df29524ca054c489efb3ae39305cb76.slice/crio-083d68b9024a7349f0bfe6d845b4d847dfb1d732c4910ebe1bca351cdf28db04 WatchSource:0}: Error finding container 083d68b9024a7349f0bfe6d845b4d847dfb1d732c4910ebe1bca351cdf28db04: Status 404 returned error can't find the container with id 083d68b9024a7349f0bfe6d845b4d847dfb1d732c4910ebe1bca351cdf28db04 Nov 03 13:03:46.481419 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.481401 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:03:46.541578 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.541516 2582 apiserver.go:52] "Watching apiserver" Nov 03 13:03:46.565697 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.565668 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 13:03:46.634371 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.634319 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"f0d4fd5bce6fe0026d1ea2d7ee6f2f03","Type":"ContainerStarted","Data":"cc857669bc04ba5000b5a232dc118f91b3248463f19cff875d13c7711cc6844d"} Nov 03 13:03:46.635284 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.635262 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"8df29524ca054c489efb3ae39305cb76","Type":"ContainerStarted","Data":"083d68b9024a7349f0bfe6d845b4d847dfb1d732c4910ebe1bca351cdf28db04"} Nov 03 13:03:46.696207 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.696182 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 13:03:46.696498 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.696481 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kf7m8","kube-system/konnectivity-agent-g6wbc","kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j","openshift-cluster-node-tuning-operator/tuned-v2pft","openshift-image-registry/node-ca-q7gwd","openshift-multus/network-metrics-daemon-ctkjc","kube-system/global-pull-secret-syncer-6ct64","openshift-multus/multus-additional-cni-plugins-l6d9s","openshift-dns/node-resolver-97s5t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal","openshift-multus/multus-t9zkj","openshift-network-diagnostics/network-check-target-nh6c9","openshift-network-operator/iptables-alerter-sdwbl"] Nov 03 13:03:46.700383 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.700368 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.702523 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.702499 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.704947 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.704930 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.707728 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.707711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.710080 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.710052 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.712350 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.712334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.714678 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.714662 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:46.714762 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.714742 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:03:46.718144 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.717134 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.718942 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.718917 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:03:46.719775 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.719757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:46.722761 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.722739 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.725020 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.725002 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:46.725127 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.725080 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:03:46.727298 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.727279 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.753157 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.753125 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 13:03:46.753157 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.753140 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g4lr6\"" Nov 03 13:03:46.753157 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.753122 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 13:03:46.753288 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.753182 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 13:03:46.753288 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.753192 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 13:03:46.753288 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.753184 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 13:03:46.754483 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.754460 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 13:03:46.754584 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.754572 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 13:03:46.754704 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.754687 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8v28q\"" Nov 03 13:03:46.756903 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.756882 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 13:03:46.756903 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.756884 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 13:03:46.757004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.756936 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 13:03:46.757004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.756953 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-85kzz\"" Nov 03 13:03:46.759759 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.759742 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 13:03:46.759827 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.759759 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 13:03:46.759916 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.759902 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 13:03:46.759966 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.759945 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lbs9x\"" Nov 03 13:03:46.760083 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.760052 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 13:03:46.760121 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.760078 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 13:03:46.760231 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.760219 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 13:03:46.766874 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.766856 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 13:03:46.776421 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-host\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.776478 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-tuned\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.776478 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-cnibin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.776478 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776461 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-etc-kubernetes\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.776570 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.776570 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776563 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-konnectivity-ca\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:46.776630 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-bin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.776630 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776618 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f645ece-f5a0-478c-a0a8-a772c740a072-iptables-alerter-script\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.776686 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.776716 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776672 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-var-lib-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.776771 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:46.776806 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.776837 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249644e9-11b8-48ad-8e70-79e3b87c550a-tmp-dir\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.776867 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btxph\" (UniqueName: \"kubernetes.io/projected/249644e9-11b8-48ad-8e70-79e3b87c550a-kube-api-access-btxph\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.776898 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776874 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-serviceca\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.776898 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776887 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-systemd-units\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.776956 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-agent-certs\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:46.776956 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776927 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-k8s-cni-cncf-io\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.776956 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-kubelet\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.777037 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.776955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysconfig\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.777037 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777027 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-kubernetes\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.777116 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777051 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-slash\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777148 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777113 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-etc-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777148 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-daemon-config\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.777201 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-multus-certs\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.777236 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-dbus\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.777266 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-netd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777313 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovn-node-metrics-cert\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777347 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777325 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.777347 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777342 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-lib-modules\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.777401 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777367 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.777401 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777391 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-device-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.777457 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777407 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtt9\" (UniqueName: \"kubernetes.io/projected/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kube-api-access-rgtt9\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.777457 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-socket-dir-parent\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.777457 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249644e9-11b8-48ad-8e70-79e3b87c550a-hosts-file\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.777543 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777459 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-sys\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.777543 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.777543 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-node-log\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777543 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7fc\" (UniqueName: \"kubernetes.io/projected/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-kube-api-access-bp7fc\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.777543 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-netns\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777543 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777536 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-systemd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-ovn\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-env-overrides\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-systemd\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.777700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-socket-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.777700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-kubelet\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777631 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-cni-binary-copy\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.777700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777667 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-cnibin\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777695 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgdd6\" (UniqueName: \"kubernetes.io/projected/6c984d62-74a3-4dab-9639-8fed18759ddc-kube-api-access-lgdd6\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777715 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-var-lib-kubelet\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777729 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-sys-fs\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-log-socket\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777779 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-script-lib\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thp2p\" (UniqueName: \"kubernetes.io/projected/dd77bb7b-1f79-4021-a9ad-70160e72cda4-kube-api-access-thp2p\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-multus\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777849 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbch\" (UniqueName: \"kubernetes.io/projected/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-kube-api-access-cvbch\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-config\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-hostroot\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.777911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777905 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-conf-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777918 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6ds8\" (UniqueName: \"kubernetes.io/projected/1f645ece-f5a0-478c-a0a8-a772c740a072-kube-api-access-s6ds8\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777932 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-tmp\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777950 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-registration-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.777993 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-netns\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f645ece-f5a0-478c-a0a8-a772c740a072-host-slash\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778051 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-host\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778082 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778105 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-modprobe-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778125 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-run\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778146 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-kubelet-config\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74697\" (UniqueName: \"kubernetes.io/projected/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-kube-api-access-74697\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-system-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.778225 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqn2s\" (UniqueName: \"kubernetes.io/projected/08ac07c7-d646-4d10-8ba3-844c752e3a37-kube-api-access-xqn2s\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.778627 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-system-cni-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.778627 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.778627 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-conf\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.778627 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.778627 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778307 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-bin\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.778627 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.778321 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:46.798353 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.798297 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 13:03:46.798522 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.798508 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 13:03:46.798849 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.798835 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 13:03:46.798890 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.798842 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 13:03:46.803525 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803504 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s6pph\"" Nov 03 13:03:46.803699 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803679 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 13:03:46.803779 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803694 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 13:03:46.803779 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803706 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 13:03:46.803779 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803755 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rxbd5" Nov 03 13:03:46.803779 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803772 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 13:03:46.803936 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803837 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5wxnl\"" Nov 03 13:03:46.803936 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803904 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:03:46.804025 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803937 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-sctm8\"" Nov 03 13:03:46.804025 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803957 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:03:46.804025 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.803973 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5ntlc\"" Nov 03 13:03:46.804135 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.804038 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l9fl5\"" Nov 03 13:03:46.804135 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.804065 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 13:03:46.804201 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.804172 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 13:03:46.879344 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-host\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.879344 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-modprobe-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-run\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-kubelet-config\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879408 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74697\" (UniqueName: \"kubernetes.io/projected/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-kube-api-access-74697\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-system-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn2s\" (UniqueName: \"kubernetes.io/projected/08ac07c7-d646-4d10-8ba3-844c752e3a37-kube-api-access-xqn2s\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-system-cni-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879486 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-kubelet-config\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879505 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-run\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.879527 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879505 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-modprobe-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879559 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-system-cni-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-conf\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879589 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-system-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-bin\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-host\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879690 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-tuned\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879696 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879757 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-host\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879785 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-cnibin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879793 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-etc-kubernetes\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879872 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-cnibin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879889 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-host\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879901 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.880093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879916 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-conf\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879931 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-etc-kubernetes\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-konnectivity-ca\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-bin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.879985 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880027 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880032 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-bin\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880016 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f645ece-f5a0-478c-a0a8-a772c740a072-iptables-alerter-script\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880115 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-bin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-var-lib-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880233 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880256 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249644e9-11b8-48ad-8e70-79e3b87c550a-tmp-dir\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880281 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btxph\" (UniqueName: \"kubernetes.io/projected/249644e9-11b8-48ad-8e70-79e3b87c550a-kube-api-access-btxph\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880298 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-var-lib-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.880895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-serviceca\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-systemd-units\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-agent-certs\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880405 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-k8s-cni-cncf-io\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880431 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-kubelet\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.880451 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysconfig\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.880513 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:03:47.380495387 +0000 UTC m=+3.173934319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880542 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysconfig\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-kubernetes\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880573 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-slash\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880576 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-konnectivity-ca\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880586 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-kubernetes\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-etc-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880664 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-daemon-config\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880689 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f645ece-f5a0-478c-a0a8-a772c740a072-iptables-alerter-script\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.881722 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880723 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249644e9-11b8-48ad-8e70-79e3b87c550a-tmp-dir\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-multus-certs\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-dbus\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880822 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-kubelet\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880834 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-k8s-cni-cncf-io\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-netd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880874 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-etc-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovn-node-metrics-cert\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880941 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-lib-modules\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.880992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881020 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-serviceca\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-device-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.881118 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtt9\" (UniqueName: \"kubernetes.io/projected/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kube-api-access-rgtt9\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881168 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-socket-dir-parent\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.881196 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret podName:eed1b9f7-3769-46a0-b8ba-63eb8d17a952 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:47.381167278 +0000 UTC m=+3.174606210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret") pod "global-pull-secret-syncer-6ct64" (UID: "eed1b9f7-3769-46a0-b8ba-63eb8d17a952") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:46.882512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881223 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-socket-dir-parent\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249644e9-11b8-48ad-8e70-79e3b87c550a-hosts-file\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-sys\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881260 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881272 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-device-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881306 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881325 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-systemd-units\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-node-log\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881339 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-netd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881359 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-node-log\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7fc\" (UniqueName: \"kubernetes.io/projected/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-kube-api-access-bp7fc\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881373 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-sys\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-netns\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-systemd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881457 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881307 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-daemon-config\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881485 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-systemd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881487 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-netns\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.883322 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881509 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-slash\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-ovn\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-env-overrides\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881549 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881553 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-ovn\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881570 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-systemd\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881576 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-lib-modules\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881619 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-socket-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881638 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-systemd\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881644 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-dbus\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-kubelet\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881682 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-multus-certs\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881724 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-kubelet\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881760 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249644e9-11b8-48ad-8e70-79e3b87c550a-hosts-file\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-cni-binary-copy\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881810 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-socket-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-cnibin\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.884057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881850 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdd6\" (UniqueName: \"kubernetes.io/projected/6c984d62-74a3-4dab-9639-8fed18759ddc-kube-api-access-lgdd6\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881951 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-cnibin\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.881986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-var-lib-kubelet\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882010 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-sys-fs\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-log-socket\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882043 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-var-lib-kubelet\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882101 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-log-socket\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-script-lib\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thp2p\" (UniqueName: \"kubernetes.io/projected/dd77bb7b-1f79-4021-a9ad-70160e72cda4-kube-api-access-thp2p\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882161 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-sys-fs\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-multus\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbch\" (UniqueName: \"kubernetes.io/projected/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-kube-api-access-cvbch\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-config\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882312 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-hostroot\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882337 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-conf-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882362 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-multus\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.884556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ds8\" (UniqueName: \"kubernetes.io/projected/1f645ece-f5a0-478c-a0a8-a772c740a072-kube-api-access-s6ds8\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882400 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-hostroot\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882553 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-env-overrides\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882594 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-tmp\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-registration-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-netns\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882785 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-netns\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882844 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882864 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-conf-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882891 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-registration-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882934 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f645ece-f5a0-478c-a0a8-a772c740a072-host-slash\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.882971 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f645ece-f5a0-478c-a0a8-a772c740a072-host-slash\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.883221 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-cni-binary-copy\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.883341 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-script-lib\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.883539 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-config\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.884050 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-tuned\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.884199 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-agent-certs\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:46.885007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.884215 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovn-node-metrics-cert\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.885487 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.884945 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-tmp\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.885487 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.885371 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:46.935857 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.935828 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:46.935973 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.935851 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:46.935973 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.935878 2582 projected.go:196] Error preparing data for projected volume kube-api-access-mhbkm for pod openshift-network-diagnostics/network-check-target-nh6c9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:46.935973 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:46.935939 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm podName:1651b048-50ba-4391-8382-60f9cffd1fa7 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:47.435924619 +0000 UTC m=+3.229363530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mhbkm" (UniqueName: "kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm") pod "network-check-target-nh6c9" (UID: "1651b048-50ba-4391-8382-60f9cffd1fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:46.938553 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.938525 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn2s\" (UniqueName: \"kubernetes.io/projected/08ac07c7-d646-4d10-8ba3-844c752e3a37-kube-api-access-xqn2s\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:03:46.938553 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.938538 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btxph\" (UniqueName: \"kubernetes.io/projected/249644e9-11b8-48ad-8e70-79e3b87c550a-kube-api-access-btxph\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:46.938825 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.938802 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74697\" (UniqueName: \"kubernetes.io/projected/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-kube-api-access-74697\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:46.939323 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.939289 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtt9\" (UniqueName: \"kubernetes.io/projected/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kube-api-access-rgtt9\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:46.939404 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.939335 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7fc\" (UniqueName: \"kubernetes.io/projected/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-kube-api-access-bp7fc\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:46.939502 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.939477 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdd6\" (UniqueName: \"kubernetes.io/projected/6c984d62-74a3-4dab-9639-8fed18759ddc-kube-api-access-lgdd6\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:46.940294 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.940275 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thp2p\" (UniqueName: \"kubernetes.io/projected/dd77bb7b-1f79-4021-a9ad-70160e72cda4-kube-api-access-thp2p\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:46.940369 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.940350 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbch\" (UniqueName: \"kubernetes.io/projected/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-kube-api-access-cvbch\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:46.941175 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:46.941159 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ds8\" (UniqueName: \"kubernetes.io/projected/1f645ece-f5a0-478c-a0a8-a772c740a072-kube-api-access-s6ds8\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:47.023700 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.023668 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-97s5t" Nov 03 13:03:47.031012 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.030988 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:03:47.033753 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.033718 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249644e9_11b8_48ad_8e70_79e3b87c550a.slice/crio-5cb6b2f6020a0a6bd4c97e359bae474d91ec5efcfce73a0bd9e5a7a250595058 WatchSource:0}: Error finding container 5cb6b2f6020a0a6bd4c97e359bae474d91ec5efcfce73a0bd9e5a7a250595058: Status 404 returned error can't find the container with id 5cb6b2f6020a0a6bd4c97e359bae474d91ec5efcfce73a0bd9e5a7a250595058 Nov 03 13:03:47.039002 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.038981 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c984d62_74a3_4dab_9639_8fed18759ddc.slice/crio-07e63683dd8aee44f4ecc697513b5cb1b6ad02fcb6eb5172555599a5df33a7ca WatchSource:0}: Error finding container 07e63683dd8aee44f4ecc697513b5cb1b6ad02fcb6eb5172555599a5df33a7ca: Status 404 returned error can't find the container with id 07e63683dd8aee44f4ecc697513b5cb1b6ad02fcb6eb5172555599a5df33a7ca Nov 03 13:03:47.041218 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.041203 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:03:47.042488 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.042469 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:03:47.045679 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.045665 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:03:47.047826 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.047809 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:47.050064 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.050041 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a6ef94_ada1_44d6_9dc4_33a3c940fcb3.slice/crio-e9fc03dda4bf0721a969454a433c5bf7f47fbeaf5212fc5344e423f6846b4fbf WatchSource:0}: Error finding container e9fc03dda4bf0721a969454a433c5bf7f47fbeaf5212fc5344e423f6846b4fbf: Status 404 returned error can't find the container with id e9fc03dda4bf0721a969454a433c5bf7f47fbeaf5212fc5344e423f6846b4fbf Nov 03 13:03:47.052872 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.052845 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d90132_40a2_4f02_8f59_e0059a1c0a1a.slice/crio-d7ce3552fe8b4800d0be4d8361aa5eb2af263395320164a1c78e49a5e35725dc WatchSource:0}: Error finding container d7ce3552fe8b4800d0be4d8361aa5eb2af263395320164a1c78e49a5e35725dc: Status 404 returned error can't find the container with id d7ce3552fe8b4800d0be4d8361aa5eb2af263395320164a1c78e49a5e35725dc Nov 03 13:03:47.056160 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.056144 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t9zkj" Nov 03 13:03:47.057166 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.057135 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7eaba1_903d_4c79_9cb9_f0ba75d8b970.slice/crio-1c90366672e0ef27d9dd4dbf0f23da3c12550a28be185bb6ac5ebb5a3b4970e2 WatchSource:0}: Error finding container 1c90366672e0ef27d9dd4dbf0f23da3c12550a28be185bb6ac5ebb5a3b4970e2: Status 404 returned error can't find the container with id 1c90366672e0ef27d9dd4dbf0f23da3c12550a28be185bb6ac5ebb5a3b4970e2 Nov 03 13:03:47.057836 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.057818 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:03:47.057925 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.057906 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6867ea1_4dd9_4ac3_8c5d_9ad6dd5dbcfc.slice/crio-873553354e91eee835b2c8273a8c51091b8df8cb25707826cfbfba6d9c35ec12 WatchSource:0}: Error finding container 873553354e91eee835b2c8273a8c51091b8df8cb25707826cfbfba6d9c35ec12: Status 404 returned error can't find the container with id 873553354e91eee835b2c8273a8c51091b8df8cb25707826cfbfba6d9c35ec12 Nov 03 13:03:47.061946 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.061865 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:03:47.064938 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.064915 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ac07c7_d646_4d10_8ba3_844c752e3a37.slice/crio-b7ba1202a628a1ad92797c9abcc8229acf4de3e7d09078afee555fbbb1288bb5 WatchSource:0}: Error finding container b7ba1202a628a1ad92797c9abcc8229acf4de3e7d09078afee555fbbb1288bb5: Status 404 returned error can't find the container with id b7ba1202a628a1ad92797c9abcc8229acf4de3e7d09078afee555fbbb1288bb5 Nov 03 13:03:47.068614 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.068568 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f645ece_f5a0_478c_a0a8_a772c740a072.slice/crio-5fabe995f2d02e05bd2f600b5ad1d727e90c8ccff06d35d24bba5eafe2f5ee48 WatchSource:0}: Error finding container 5fabe995f2d02e05bd2f600b5ad1d727e90c8ccff06d35d24bba5eafe2f5ee48: Status 404 returned error can't find the container with id 5fabe995f2d02e05bd2f600b5ad1d727e90c8ccff06d35d24bba5eafe2f5ee48 Nov 03 13:03:47.071029 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:03:47.071012 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd77bb7b_1f79_4021_a9ad_70160e72cda4.slice/crio-96ea355720fe72a5adc432280a5160a0bd7fc9c5c79c5f89c8171c358556ed91 WatchSource:0}: Error finding container 96ea355720fe72a5adc432280a5160a0bd7fc9c5c79c5f89c8171c358556ed91: Status 404 returned error can't find the container with id 96ea355720fe72a5adc432280a5160a0bd7fc9c5c79c5f89c8171c358556ed91 Nov 03 13:03:47.104169 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.104149 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:47.385220 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.385138 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:47.385385 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.385224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:47.385385 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:47.385287 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:47.385385 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:47.385328 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:47.385385 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:47.385345 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret podName:eed1b9f7-3769-46a0-b8ba-63eb8d17a952 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:48.385332078 +0000 UTC m=+4.178770993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret") pod "global-pull-secret-syncer-6ct64" (UID: "eed1b9f7-3769-46a0-b8ba-63eb8d17a952") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:47.385385 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:47.385374 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:03:48.385360891 +0000 UTC m=+4.178799817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:47.486376 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.486340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:47.486539 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:47.486522 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:47.486580 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:47.486543 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:47.486580 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:47.486553 2582 projected.go:196] Error preparing data for projected volume kube-api-access-mhbkm for pod openshift-network-diagnostics/network-check-target-nh6c9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:47.486642 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:47.486605 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm podName:1651b048-50ba-4391-8382-60f9cffd1fa7 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:48.48659164 +0000 UTC m=+4.280030565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhbkm" (UniqueName: "kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm") pod "network-check-target-nh6c9" (UID: "1651b048-50ba-4391-8382-60f9cffd1fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:47.638673 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.638590 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6wbc" event={"ID":"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc","Type":"ContainerStarted","Data":"873553354e91eee835b2c8273a8c51091b8df8cb25707826cfbfba6d9c35ec12"} Nov 03 13:03:47.640261 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.640232 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q7gwd" event={"ID":"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970","Type":"ContainerStarted","Data":"1c90366672e0ef27d9dd4dbf0f23da3c12550a28be185bb6ac5ebb5a3b4970e2"} Nov 03 13:03:47.641770 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.641740 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" event={"ID":"6c984d62-74a3-4dab-9639-8fed18759ddc","Type":"ContainerStarted","Data":"07e63683dd8aee44f4ecc697513b5cb1b6ad02fcb6eb5172555599a5df33a7ca"} Nov 03 13:03:47.643140 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.643118 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97s5t" event={"ID":"249644e9-11b8-48ad-8e70-79e3b87c550a","Type":"ContainerStarted","Data":"5cb6b2f6020a0a6bd4c97e359bae474d91ec5efcfce73a0bd9e5a7a250595058"} Nov 03 13:03:47.644906 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.644881 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"96ea355720fe72a5adc432280a5160a0bd7fc9c5c79c5f89c8171c358556ed91"} Nov 03 13:03:47.646377 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.646353 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sdwbl" event={"ID":"1f645ece-f5a0-478c-a0a8-a772c740a072","Type":"ContainerStarted","Data":"5fabe995f2d02e05bd2f600b5ad1d727e90c8ccff06d35d24bba5eafe2f5ee48"} Nov 03 13:03:47.647737 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.647711 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t9zkj" event={"ID":"08ac07c7-d646-4d10-8ba3-844c752e3a37","Type":"ContainerStarted","Data":"b7ba1202a628a1ad92797c9abcc8229acf4de3e7d09078afee555fbbb1288bb5"} Nov 03 13:03:47.649270 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.649246 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v2pft" event={"ID":"d8d90132-40a2-4f02-8f59-e0059a1c0a1a","Type":"ContainerStarted","Data":"d7ce3552fe8b4800d0be4d8361aa5eb2af263395320164a1c78e49a5e35725dc"} Nov 03 13:03:47.650511 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.650492 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" event={"ID":"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3","Type":"ContainerStarted","Data":"e9fc03dda4bf0721a969454a433c5bf7f47fbeaf5212fc5344e423f6846b4fbf"} Nov 03 13:03:47.807427 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.807377 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:58:46 +0000 UTC" deadline="2028-06-08 19:25:23.875953999 +0000 UTC" Nov 03 13:03:47.807427 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:47.807422 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13998h21m36.068535417s" Nov 03 13:03:48.392969 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:48.392933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:48.393180 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:48.393027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:48.393247 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.393182 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:48.393299 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.393246 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:03:50.393227953 +0000 UTC m=+6.186666869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:48.393410 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.393305 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:48.393410 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.393360 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret podName:eed1b9f7-3769-46a0-b8ba-63eb8d17a952 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:50.3933281 +0000 UTC m=+6.186767017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret") pod "global-pull-secret-syncer-6ct64" (UID: "eed1b9f7-3769-46a0-b8ba-63eb8d17a952") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:48.494115 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:48.494000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:48.494276 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.494174 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:48.494276 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.494193 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:48.494276 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.494206 2582 projected.go:196] Error preparing data for projected volume kube-api-access-mhbkm for pod openshift-network-diagnostics/network-check-target-nh6c9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:48.494276 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.494264 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm podName:1651b048-50ba-4391-8382-60f9cffd1fa7 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:50.494246508 +0000 UTC m=+6.287685422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhbkm" (UniqueName: "kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm") pod "network-check-target-nh6c9" (UID: "1651b048-50ba-4391-8382-60f9cffd1fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:48.632776 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:48.632737 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:48.632948 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.632882 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:03:48.633029 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:48.632959 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:48.633029 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.633022 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:03:48.633152 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:48.633092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:48.633219 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:48.633196 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:03:48.808547 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:48.808457 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:58:46 +0000 UTC" deadline="2028-04-24 16:01:54.929702191 +0000 UTC" Nov 03 13:03:48.808547 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:48.808492 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12914h58m6.121214931s" Nov 03 13:03:50.408720 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:50.408683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:50.409246 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:50.408783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:50.409246 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.408919 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:50.409246 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.408980 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:03:54.408962262 +0000 UTC m=+10.202401191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:50.409428 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.409415 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:50.409479 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.409469 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret podName:eed1b9f7-3769-46a0-b8ba-63eb8d17a952 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:54.40945433 +0000 UTC m=+10.202893243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret") pod "global-pull-secret-syncer-6ct64" (UID: "eed1b9f7-3769-46a0-b8ba-63eb8d17a952") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:50.511386 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:50.509410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:50.511386 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.509543 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:50.511386 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.509561 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:50.511386 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.509574 2582 projected.go:196] Error preparing data for projected volume kube-api-access-mhbkm for pod openshift-network-diagnostics/network-check-target-nh6c9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:50.511386 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.509623 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm podName:1651b048-50ba-4391-8382-60f9cffd1fa7 nodeName:}" failed. No retries permitted until 2026-11-03 13:03:54.509607286 +0000 UTC m=+10.303046211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhbkm" (UniqueName: "kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm") pod "network-check-target-nh6c9" (UID: "1651b048-50ba-4391-8382-60f9cffd1fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:50.633217 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:50.632431 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:50.633343 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:50.632671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:50.633456 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.633425 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:03:50.633566 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:50.632696 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:50.633566 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.633535 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:03:50.634725 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:50.634681 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:03:50.683809 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:50.683769 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"8df29524ca054c489efb3ae39305cb76","Type":"ContainerStarted","Data":"2b5f28c8e66cb821be25f88faef4781dde629c3f77d1ffffcb48e7a248bfdc85"} Nov 03 13:03:51.693309 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:51.693278 2582 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="7d3f9925cc596e905c0be838f59e0a31ac804d5ae072fc9d28fff169578b2cf2" exitCode=0 Nov 03 13:03:51.693784 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:51.693352 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" event={"ID":"6c984d62-74a3-4dab-9639-8fed18759ddc","Type":"ContainerDied","Data":"7d3f9925cc596e905c0be838f59e0a31ac804d5ae072fc9d28fff169578b2cf2"} Nov 03 13:03:51.699743 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:51.699696 2582 generic.go:356] "Generic (PLEG): container finished" podID="f0d4fd5bce6fe0026d1ea2d7ee6f2f03" containerID="2fd266aa8699820e818da5056bb5b95f5edae66484b63281f1eb2a81fd89b9e1" exitCode=0 Nov 03 13:03:51.699864 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:51.699837 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"f0d4fd5bce6fe0026d1ea2d7ee6f2f03","Type":"ContainerDied","Data":"2fd266aa8699820e818da5056bb5b95f5edae66484b63281f1eb2a81fd89b9e1"} Nov 03 13:03:51.916298 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:51.915176 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" podStartSLOduration=5.915162388 podStartE2EDuration="5.915162388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:50.798498817 +0000 UTC m=+6.591937753" watchObservedRunningTime="2026-11-03 13:03:51.915162388 +0000 UTC m=+7.708601370" Nov 03 13:03:52.630129 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:52.630092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:52.630297 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:52.630235 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:03:52.630656 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:52.630634 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:52.630776 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:52.630750 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:03:52.630840 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:52.630827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:52.630943 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:52.630924 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:03:52.705906 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:52.705332 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"f0d4fd5bce6fe0026d1ea2d7ee6f2f03","Type":"ContainerStarted","Data":"99b8db5f8d4c2c2dfafcc4174e78edace193eed84a541d7d2e271d603721b4b8"} Nov 03 13:03:52.710125 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:52.710043 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6wbc" event={"ID":"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc","Type":"ContainerStarted","Data":"4e702768c2ef2a1ca0d19464156ddeae45ffbc34629d3a132623974742567fd0"} Nov 03 13:03:52.933871 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:52.933812 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g6wbc" podStartSLOduration=3.290521194 podStartE2EDuration="7.933792511s" totalImagesPullingTime="4.643271317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:52.933691601 +0000 UTC m=+8.727130537" watchObservedRunningTime="2026-11-03 13:03:52.933792511 +0000 UTC m=+8.727231447" Nov 03 13:03:52.934376 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:52.934342 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" podStartSLOduration=6.934330664 podStartE2EDuration="6.934330664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:52.820207058 +0000 UTC m=+8.613645992" watchObservedRunningTime="2026-11-03 13:03:52.934330664 +0000 UTC m=+8.727769598" Nov 03 13:03:53.717025 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:53.716986 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" event={"ID":"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3","Type":"ContainerStarted","Data":"1521a2a8aee0e3be740329df97d65aab95a9e751c3313df6358e31accba1bdd9"} Nov 03 13:03:53.720819 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:53.720794 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q7gwd" event={"ID":"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970","Type":"ContainerStarted","Data":"7a75b1dbe7a992dc61e3e3e80986b2573a8b9a2d7504d4280c9b0f812bd39cda"} Nov 03 13:03:54.447580 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:54.447540 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:54.447757 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:54.447607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:54.447823 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.447774 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:54.447873 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.447840 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret podName:eed1b9f7-3769-46a0-b8ba-63eb8d17a952 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:02.447822276 +0000 UTC m=+18.241261191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret") pod "global-pull-secret-syncer-6ct64" (UID: "eed1b9f7-3769-46a0-b8ba-63eb8d17a952") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:54.448048 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.448030 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:54.448178 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.448097 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:04:02.448081295 +0000 UTC m=+18.241520221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:54.548424 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:54.548387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:54.548565 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.548532 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:54.548565 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.548549 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:54.548565 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.548562 2582 projected.go:196] Error preparing data for projected volume kube-api-access-mhbkm for pod openshift-network-diagnostics/network-check-target-nh6c9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:54.548661 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.548617 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm podName:1651b048-50ba-4391-8382-60f9cffd1fa7 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:02.548601425 +0000 UTC m=+18.342040343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhbkm" (UniqueName: "kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm") pod "network-check-target-nh6c9" (UID: "1651b048-50ba-4391-8382-60f9cffd1fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:54.632504 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:54.631935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:54.632504 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:54.632016 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:54.632504 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:54.632036 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:54.632504 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.632156 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:03:54.632504 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.632187 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:03:54.632504 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:54.632243 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:03:55.386243 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:55.386206 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:55.387080 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:55.387043 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:55.502042 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:55.501989 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q7gwd" podStartSLOduration=4.717216564 podStartE2EDuration="10.501970748s" totalImagesPullingTime="5.784754184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:53.834426938 +0000 UTC m=+9.627865873" watchObservedRunningTime="2026-11-03 13:03:55.501970748 +0000 UTC m=+11.295409684" Nov 03 13:03:55.733097 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:55.730823 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-97s5t" event={"ID":"249644e9-11b8-48ad-8e70-79e3b87c550a","Type":"ContainerStarted","Data":"2d8e6ac07df0736d8d9ac557d6ba79250329c09c3f615b39809c775012052246"} Nov 03 13:03:55.733709 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:55.733678 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t9zkj" event={"ID":"08ac07c7-d646-4d10-8ba3-844c752e3a37","Type":"ContainerStarted","Data":"aa9e8295e2d0849fdea1d0e403a4bcd1ca4ecd72b816e699612a3e91efaf7528"} Nov 03 13:03:55.891257 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:55.891227 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 13:03:55.960137 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:55.960059 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-97s5t" podStartSLOduration=3.498130538 podStartE2EDuration="11.960041078s" totalImagesPullingTime="8.46191054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:55.845016201 +0000 UTC m=+11.638455135" watchObservedRunningTime="2026-11-03 13:03:55.960041078 +0000 UTC m=+11.753480013" Nov 03 13:03:56.631224 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.631146 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:56.631787 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:56.631258 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:03:56.631787 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.631594 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:56.631787 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:56.631663 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:03:56.631787 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.631786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:56.631994 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:56.631838 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:03:56.739104 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.739053 2582 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="5ac0adfadcfa82314a6ceec07ba61355eef0c67b440caa0014af0cab44266eb7" exitCode=0 Nov 03 13:03:56.739249 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.739110 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" event={"ID":"6c984d62-74a3-4dab-9639-8fed18759ddc","Type":"ContainerDied","Data":"5ac0adfadcfa82314a6ceec07ba61355eef0c67b440caa0014af0cab44266eb7"} Nov 03 13:03:56.743571 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.743045 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" event={"ID":"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3","Type":"ContainerStarted","Data":"bbafd6dddd9bfe0f8cc87145f56e8dd0268442696a00d929313d64d113dbde4f"} Nov 03 13:03:56.775830 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.775729 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:03:55.891255778Z","UUID":"f2d5634f-bd93-4399-bbae-99a0b30723b4","Handler":null,"Name":"","Endpoint":""} Nov 03 13:03:56.777801 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.777586 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 13:03:56.777918 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.777811 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 13:03:56.857572 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:56.857512 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-t9zkj" podStartSLOduration=3.328851271 podStartE2EDuration="11.857492012s" totalImagesPullingTime="8.528640741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:55.961204789 +0000 UTC m=+11.754643724" watchObservedRunningTime="2026-11-03 13:03:56.857492012 +0000 UTC m=+12.650930949" Nov 03 13:03:57.747478 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:57.747039 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sdwbl" event={"ID":"1f645ece-f5a0-478c-a0a8-a772c740a072","Type":"ContainerStarted","Data":"a58d46e2f9b8dfd009cf5096c73a3c77c26ed8858a330ebb6250f918e690d3b9"} Nov 03 13:03:57.860965 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:57.860911 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sdwbl" podStartSLOduration=4.1795397340000005 podStartE2EDuration="12.860895049s" totalImagesPullingTime="8.681355315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:57.860802216 +0000 UTC m=+13.654241158" watchObservedRunningTime="2026-11-03 13:03:57.860895049 +0000 UTC m=+13.654333984" Nov 03 13:03:57.957254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:57.957215 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:57.957433 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:57.957370 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:03:57.957844 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:57.957817 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:03:58.633496 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:58.633468 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:03:58.633607 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:58.633503 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:03:58.633673 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:58.633650 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:03:58.633752 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:58.633693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:03:58.633802 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:58.633765 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:03:58.633851 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:03:58.633840 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:03:58.751282 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:58.751183 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v2pft" event={"ID":"d8d90132-40a2-4f02-8f59-e0059a1c0a1a","Type":"ContainerStarted","Data":"9688d28dddd986ef12a38ad9c1ab0906242b581defb12f3af749ab0d351c7d98"} Nov 03 13:03:58.754126 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:58.754092 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" event={"ID":"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3","Type":"ContainerStarted","Data":"a21a376348255d0b6d0dc934d7f7009a9f0568202f88e41749e274cbc2056b6a"} Nov 03 13:03:58.756475 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:58.756444 2582 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="28c64cd91bfb828140ef16887211d38e710476a20453e2d35c232e5c43526658" exitCode=0 Nov 03 13:03:58.756620 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:58.756501 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" event={"ID":"6c984d62-74a3-4dab-9639-8fed18759ddc","Type":"ContainerDied","Data":"28c64cd91bfb828140ef16887211d38e710476a20453e2d35c232e5c43526658"} Nov 03 13:03:58.866000 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:58.865947 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v2pft" podStartSLOduration=2.8766961760000003 podStartE2EDuration="13.865930689s" totalImagesPullingTime="10.989234513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:58.865307386 +0000 UTC m=+14.658746320" watchObservedRunningTime="2026-11-03 13:03:58.865930689 +0000 UTC m=+14.659369623" Nov 03 13:03:59.103916 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:03:59.103800 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" podStartSLOduration=3.532526635 podStartE2EDuration="14.103782521s" totalImagesPullingTime="10.571255886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:59.103097648 +0000 UTC m=+14.896536583" watchObservedRunningTime="2026-11-03 13:03:59.103782521 +0000 UTC m=+14.897221455" Nov 03 13:04:00.630186 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:00.630101 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:00.630186 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:00.630132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:00.630911 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:00.630099 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:00.630911 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:00.630238 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:04:00.630911 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:00.630313 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:04:00.630911 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:00.630393 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:00.764100 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:00.764054 2582 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="3a054f99789efdaca945d0668cc0d5bff33a7e19d70f2040d39101e5bec37312" exitCode=0 Nov 03 13:04:00.764266 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:00.764126 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" event={"ID":"6c984d62-74a3-4dab-9639-8fed18759ddc","Type":"ContainerDied","Data":"3a054f99789efdaca945d0668cc0d5bff33a7e19d70f2040d39101e5bec37312"} Nov 03 13:04:02.510817 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.510546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:02.510817 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.510770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:02.510817 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.510692 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:02.511291 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.510873 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:04:18.510849729 +0000 UTC m=+34.304288654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:02.511291 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.510886 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:02.511291 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.510940 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret podName:eed1b9f7-3769-46a0-b8ba-63eb8d17a952 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:18.510928371 +0000 UTC m=+34.304367296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret") pod "global-pull-secret-syncer-6ct64" (UID: "eed1b9f7-3769-46a0-b8ba-63eb8d17a952") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:02.611605 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.611580 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:02.611776 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.611758 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:02.611835 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.611782 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:02.611835 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.611792 2582 projected.go:196] Error preparing data for projected volume kube-api-access-mhbkm for pod openshift-network-diagnostics/network-check-target-nh6c9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:02.611904 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.611839 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm podName:1651b048-50ba-4391-8382-60f9cffd1fa7 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:18.611826422 +0000 UTC m=+34.405265334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhbkm" (UniqueName: "kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm") pod "network-check-target-nh6c9" (UID: "1651b048-50ba-4391-8382-60f9cffd1fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:02.633798 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.633752 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:02.633798 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.633775 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:02.634007 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.633775 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:02.634007 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.633875 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:02.634007 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.633985 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:04:02.634156 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:02.634088 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:04:02.771928 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.771889 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"76a78da3aecdb3781c7e9f6805d394f8e8f704ac4a8ef4aff8c69dc8cce6371c"} Nov 03 13:04:02.772063 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.771972 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"34f8c8a06df1b27e1db7cd9b56968efdd80e411a9367eed63f536072cc2c54b7"} Nov 03 13:04:02.772063 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.772012 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"9e12572578fac31d8fa87dd117ffe175e5f695a1249cbdc393a93bfee03f9647"} Nov 03 13:04:02.772063 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.772049 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"06b2122ee8597f19e0861f2ebf6b91bc693483445e12fca7562ab4dcbf74ce92"} Nov 03 13:04:02.772181 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:02.772112 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"7acf980d2bfe898b6f3d336994a35786ff2d0f7dbfbe9d4709d18435f1b3a5e8"} Nov 03 13:04:03.776956 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:03.776923 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"08ac582465c865b4e6113b3c78b180bbb4694ca07c186e3091fcf29d1bc34d9a"} Nov 03 13:04:03.779432 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:03.779408 2582 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="af0d55183b8184a93e9f93cd1c203680f3e9fa12c7556bd29f350cbe462308e9" exitCode=0 Nov 03 13:04:03.779549 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:03.779446 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" event={"ID":"6c984d62-74a3-4dab-9639-8fed18759ddc","Type":"ContainerDied","Data":"af0d55183b8184a93e9f93cd1c203680f3e9fa12c7556bd29f350cbe462308e9"} Nov 03 13:04:04.630788 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:04.630749 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:04.630930 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:04.630839 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:04.630930 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:04.630917 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:04.633841 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:04.631460 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:04:04.633841 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:04.631559 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:04.633841 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:04.631977 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:04:04.783578 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:04.783540 2582 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="b72b2e7663ee6c9e70f4747af575b5c3639ba5c71bd5859936628de28e58047a" exitCode=0 Nov 03 13:04:04.784020 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:04.783607 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" event={"ID":"6c984d62-74a3-4dab-9639-8fed18759ddc","Type":"ContainerDied","Data":"b72b2e7663ee6c9e70f4747af575b5c3639ba5c71bd5859936628de28e58047a"} Nov 03 13:04:05.788517 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:05.788484 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"d4eecbecfa567fb12c6dbd78bb52a3af32ab209124c00d8194026b99bbc51a2e"} Nov 03 13:04:05.791501 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:05.791474 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" event={"ID":"6c984d62-74a3-4dab-9639-8fed18759ddc","Type":"ContainerStarted","Data":"a5b4c3da29df2b17307539fcbc598e5ce100d3cf1a2d823ae17846bd70df7e43"} Nov 03 13:04:05.910559 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:05.910509 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l6d9s" podStartSLOduration=8.424099905 podStartE2EDuration="21.910495774s" totalImagesPullingTime="13.486395869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:05.909958809 +0000 UTC m=+21.703397754" watchObservedRunningTime="2026-11-03 13:04:05.910495774 +0000 UTC m=+21.703934708" Nov 03 13:04:06.630324 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:06.630288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:06.630324 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:06.630322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:06.630524 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:06.630289 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:06.630524 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:06.630403 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:04:06.630524 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:06.630476 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:04:06.630605 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:06.630560 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:07.799172 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:07.799131 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" event={"ID":"dd77bb7b-1f79-4021-a9ad-70160e72cda4","Type":"ContainerStarted","Data":"b02f9bd0bbd8ba6fef1e9fefac038524ed0205c43da44747334b6e1131b7145a"} Nov 03 13:04:07.799857 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:07.799340 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:07.814639 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:07.814494 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:08.073177 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:08.073041 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" podStartSLOduration=7.703750542 podStartE2EDuration="23.073029654s" totalImagesPullingTime="15.369279112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:08.02219254 +0000 UTC m=+23.815631474" watchObservedRunningTime="2026-11-03 13:04:08.073029654 +0000 UTC m=+23.866468588" Nov 03 13:04:08.630292 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:08.630258 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:08.630553 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:08.630327 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:08.630553 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:08.630330 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:08.630553 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:08.630424 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:04:08.630553 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:08.630518 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:08.630729 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:08.630601 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:04:08.801872 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:08.801848 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:08.802226 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:08.801909 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:08.816342 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:08.816312 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:09.527043 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:09.527012 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ct64"] Nov 03 13:04:09.527214 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:09.527182 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:09.527309 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:09.527277 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:09.530181 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:09.530157 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ctkjc"] Nov 03 13:04:09.530285 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:09.530273 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:09.530401 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:09.530384 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:04:09.530757 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:09.530741 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nh6c9"] Nov 03 13:04:09.530822 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:09.530812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:09.530887 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:09.530874 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:04:10.630645 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:10.630609 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:10.631017 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:10.630743 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:11.630444 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:11.630406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:11.630595 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:11.630516 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:04:11.630595 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:11.630574 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:11.630962 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:11.630705 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:04:12.630557 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:12.630521 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:12.630791 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:12.630681 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:13.629977 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:13.629901 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:13.630131 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:13.629901 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:13.630131 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:13.630035 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctkjc" podUID="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" Nov 03 13:04:13.630199 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:13.630144 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh6c9" podUID="1651b048-50ba-4391-8382-60f9cffd1fa7" Nov 03 13:04:14.631343 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:14.631308 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:14.631728 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:14.631427 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6ct64" podUID="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" Nov 03 13:04:15.526968 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.526945 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeReady" Nov 03 13:04:15.527146 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.527054 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 13:04:15.630612 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.630579 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:15.630765 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.630582 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:15.640359 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.640338 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f6bgd"] Nov 03 13:04:15.643311 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.643293 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-67jgd"] Nov 03 13:04:15.643435 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.643419 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" Nov 03 13:04:15.646049 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.646032 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w6sm7"] Nov 03 13:04:15.646512 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.646486 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:15.650248 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.650226 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" Nov 03 13:04:15.681703 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.680921 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-92c7q"] Nov 03 13:04:15.683958 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.683943 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv"] Nov 03 13:04:15.684093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.684062 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-92c7q" Nov 03 13:04:15.686627 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.686610 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.690624 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.690608 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs"] Nov 03 13:04:15.693435 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.693404 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2pfz7"] Nov 03 13:04:15.693720 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.693702 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" Nov 03 13:04:15.696318 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.696299 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w"] Nov 03 13:04:15.696451 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.696433 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.699235 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.699216 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp"] Nov 03 13:04:15.699354 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.699339 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.701837 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.701824 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk"] Nov 03 13:04:15.701934 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.701922 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.704810 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.704792 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8478467d49-l5dtm"] Nov 03 13:04:15.704894 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.704882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.707644 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.707631 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5564c5bd47-g6p67"] Nov 03 13:04:15.707784 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.707767 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.710284 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.710266 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xhtjf"] Nov 03 13:04:15.710592 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.710399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.713750 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.713732 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr"] Nov 03 13:04:15.713841 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.713773 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.718583 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.718567 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" Nov 03 13:04:15.730660 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.730639 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.730771 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.730639 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.730771 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.730703 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:04:15.731431 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.731411 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-896bc546-4fp47"] Nov 03 13:04:15.734704 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.734688 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hp7ng"] Nov 03 13:04:15.734870 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.734855 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.739402 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.739315 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k6d8j"] Nov 03 13:04:15.743086 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.740369 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nbm\"" Nov 03 13:04:15.743086 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.740485 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pvm6f\"" Nov 03 13:04:15.745358 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.745334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:15.751088 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.751048 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f6bgd"] Nov 03 13:04:15.751194 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.751096 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w6sm7"] Nov 03 13:04:15.751194 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.751174 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:15.752141 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.752119 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-92c7q"] Nov 03 13:04:15.752709 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.752690 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv"] Nov 03 13:04:15.753896 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.753873 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs"] Nov 03 13:04:15.754057 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.754044 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w"] Nov 03 13:04:15.754769 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.754749 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp"] Nov 03 13:04:15.755385 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.755366 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2pfz7"] Nov 03 13:04:15.756160 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.756138 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5564c5bd47-g6p67"] Nov 03 13:04:15.756739 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.756722 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xhtjf"] Nov 03 13:04:15.760046 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.760024 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-896bc546-4fp47"] Nov 03 13:04:15.787576 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.787524 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.787706 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.787631 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 13:04:15.787776 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.787702 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9xv7j\"" Nov 03 13:04:15.787852 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.787837 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fg5qt\"" Nov 03 13:04:15.787909 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.787895 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.788110 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.788066 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 13:04:15.788202 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.788135 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.788202 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.788174 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5prrc\"" Nov 03 13:04:15.788305 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.788215 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.788305 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.788220 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 13:04:15.788400 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.788309 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-697z8\"" Nov 03 13:04:15.788400 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.788350 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 13:04:15.788400 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.788372 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 13:04:15.790885 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.790864 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 13:04:15.790963 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.790917 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.791022 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.790991 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.791091 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.791050 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.791239 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.791222 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.791346 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.791255 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.791404 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.791294 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.791404 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.791363 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 13:04:15.791404 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.791307 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 13:04:15.791404 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.791386 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.793920 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.793904 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 13:04:15.793996 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.793924 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.795549 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.795533 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 13:04:15.795603 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.795529 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.795641 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.795598 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 13:04:15.798082 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.798053 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.798340 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.798324 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dh6vm\"" Nov 03 13:04:15.798409 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.798393 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bsb9h\"" Nov 03 13:04:15.798458 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.798449 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.806252 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.806362 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.806362 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806309 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2zf7\" (UniqueName: \"kubernetes.io/projected/7d1d0d87-bbca-4424-bfad-d0b312732700-kube-api-access-s2zf7\") pod \"network-check-source-6f78d59f99-4nxqr\" (UID: \"7d1d0d87-bbca-4424-bfad-d0b312732700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" Nov 03 13:04:15.806457 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccn5\" (UniqueName: \"kubernetes.io/projected/ec0b3420-306f-4a10-95ff-0bcf4a4f1730-kube-api-access-hccn5\") pod \"volume-data-source-validator-58c9f5f46d-x6jrs\" (UID: \"ec0b3420-306f-4a10-95ff-0bcf4a4f1730\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" Nov 03 13:04:15.806494 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-stats-auth\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.806526 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-snapshots\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.806556 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806527 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjwcb\" (UniqueName: \"kubernetes.io/projected/53c4e88e-1203-4083-98e9-9a2202a3a704-kube-api-access-kjwcb\") pod \"kas-connection-checker-685895c7c9-92c7q\" (UID: \"53c4e88e-1203-4083-98e9-9a2202a3a704\") " pod="kube-system/kas-connection-checker-685895c7c9-92c7q" Nov 03 13:04:15.806588 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.806588 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71f66428-6b35-45cc-ac46-ce5c07aa958c-config\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.806662 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:15.806662 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71f66428-6b35-45cc-ac46-ce5c07aa958c-serving-cert\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.806662 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rnmt\" (UniqueName: \"kubernetes.io/projected/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-kube-api-access-4rnmt\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.806769 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:15.806769 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806718 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.806769 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.806906 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszmj\" (UniqueName: \"kubernetes.io/projected/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-kube-api-access-gszmj\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.806906 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806797 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.806906 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-serving-cert\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.806906 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.806906 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806877 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.807041 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806905 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.807041 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806929 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfgn\" (UniqueName: \"kubernetes.io/projected/05516dd7-e72f-4b64-9707-e401c7e6198b-kube-api-access-cpfgn\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.807041 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f70a3801-ffa0-4049-aeee-0d886ccb30b3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.807041 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.806988 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.807041 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807014 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50d1326a-e89d-4e45-90c6-9a254a157e48-serving-cert\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.807254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.807254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807063 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.807254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.807254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807133 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/f70a3801-ffa0-4049-aeee-0d886ccb30b3-kube-api-access-dl2z6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.807254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.807254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/541aac48-8aa9-40e2-88ec-17687fdd5167-cert\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:15.807254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807213 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79g67\" (UniqueName: \"kubernetes.io/projected/50d1326a-e89d-4e45-90c6-9a254a157e48-kube-api-access-79g67\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.807254 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-trusted-ca\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.807469 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.807469 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807301 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.807469 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.807469 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807358 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-tmp\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.807469 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2289v\" (UniqueName: \"kubernetes.io/projected/6365006c-9e6f-406f-9102-e3b0b452675d-kube-api-access-2289v\") pod \"kas-connection-checker-685895c7c9-w6sm7\" (UID: \"6365006c-9e6f-406f-9102-e3b0b452675d\") " pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" Nov 03 13:04:15.807469 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807411 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qc5\" (UniqueName: \"kubernetes.io/projected/71f66428-6b35-45cc-ac46-ce5c07aa958c-kube-api-access-67qc5\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.807469 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807436 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f70a3801-ffa0-4049-aeee-0d886ccb30b3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.807469 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807465 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.807741 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-config\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.807741 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.807741 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807549 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.807741 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807580 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfds\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.807741 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-default-certificate\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.807741 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfz2t\" (UniqueName: \"kubernetes.io/projected/541aac48-8aa9-40e2-88ec-17687fdd5167-kube-api-access-rfz2t\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:15.807741 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trds9\" (UniqueName: \"kubernetes.io/projected/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70-kube-api-access-trds9\") pod \"kas-connection-checker-685895c7c9-f6bgd\" (UID: \"78248e2d-8c0b-47fb-b55b-07e0ffdc1c70\") " pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" Nov 03 13:04:15.807741 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807730 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q822s\" (UniqueName: \"kubernetes.io/projected/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-kube-api-access-q822s\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.807981 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807764 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzwn\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.807981 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.807981 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.807838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.810028 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.810003 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.810028 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.810018 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-brvx7\"" Nov 03 13:04:15.810028 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.810028 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.810204 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.810059 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 13:04:15.838357 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838326 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 13:04:15.838357 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838349 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 13:04:15.838562 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838389 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 13:04:15.838562 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838441 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t56mb\"" Nov 03 13:04:15.838562 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838515 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.838562 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838546 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 13:04:15.838562 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838517 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7pkcc\"" Nov 03 13:04:15.838766 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838542 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 13:04:15.838766 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838597 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 13:04:15.838766 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838645 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 13:04:15.838766 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838647 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 13:04:15.838766 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.838703 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 13:04:15.841296 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.841278 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4m6js\"" Nov 03 13:04:15.848877 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.848862 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 13:04:15.851709 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.851693 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-67jgd"] Nov 03 13:04:15.860607 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.860588 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 13:04:15.864867 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.861674 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 13:04:15.864867 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.864855 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk"] Nov 03 13:04:15.865547 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.865532 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr"] Nov 03 13:04:15.888854 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.888722 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 13:04:15.888854 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.888789 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 13:04:15.888854 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.888798 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 13:04:15.888854 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.888801 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 13:04:15.888854 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.888790 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 13:04:15.888854 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.888838 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9jvp2\"" Nov 03 13:04:15.888854 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.888859 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m8wc5\"" Nov 03 13:04:15.889320 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.889305 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8478467d49-l5dtm"] Nov 03 13:04:15.889883 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.889873 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k6d8j"] Nov 03 13:04:15.890439 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.890413 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hp7ng"] Nov 03 13:04:15.909132 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f70a3801-ffa0-4049-aeee-0d886ccb30b3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.909238 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.909238 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50d1326a-e89d-4e45-90c6-9a254a157e48-serving-cert\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.909238 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.909378 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.909378 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909335 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.909378 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.909363 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:15.909518 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.909446 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:16.409424122 +0000 UTC m=+32.202863057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:15.909518 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909366 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/f70a3801-ffa0-4049-aeee-0d886ccb30b3-kube-api-access-dl2z6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.909518 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.909665 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/541aac48-8aa9-40e2-88ec-17687fdd5167-cert\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:15.909762 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909743 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.909868 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909843 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f70a3801-ffa0-4049-aeee-0d886ccb30b3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.909948 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909925 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79g67\" (UniqueName: \"kubernetes.io/projected/50d1326a-e89d-4e45-90c6-9a254a157e48-kube-api-access-79g67\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.910005 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.909977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-trusted-ca\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.910054 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.910054 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910033 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.910179 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.910179 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.910179 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.910158 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:15.910179 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.910173 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:15.910371 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.910217 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:16.410201669 +0000 UTC m=+32.203640597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:15.910371 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-tmp\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.910371 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910251 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.910371 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2289v\" (UniqueName: \"kubernetes.io/projected/6365006c-9e6f-406f-9102-e3b0b452675d-kube-api-access-2289v\") pod \"kas-connection-checker-685895c7c9-w6sm7\" (UID: \"6365006c-9e6f-406f-9102-e3b0b452675d\") " pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" Nov 03 13:04:15.910371 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:15.910371 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qc5\" (UniqueName: \"kubernetes.io/projected/71f66428-6b35-45cc-ac46-ce5c07aa958c-kube-api-access-67qc5\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.910371 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910337 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f70a3801-ffa0-4049-aeee-0d886ccb30b3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.910371 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910356 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae820d-6217-425b-9451-247f5dff70dc-tmp-dir\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-config\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910471 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfds\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-default-certificate\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910518 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfz2t\" (UniqueName: \"kubernetes.io/projected/541aac48-8aa9-40e2-88ec-17687fdd5167-kube-api-access-rfz2t\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910532 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-tmp\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trds9\" (UniqueName: \"kubernetes.io/projected/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70-kube-api-access-trds9\") pod \"kas-connection-checker-685895c7c9-f6bgd\" (UID: \"78248e2d-8c0b-47fb-b55b-07e0ffdc1c70\") " pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910577 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q822s\" (UniqueName: \"kubernetes.io/projected/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-kube-api-access-q822s\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzwn\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.910767 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910624 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9htq\" (UniqueName: \"kubernetes.io/projected/99ae820d-6217-425b-9451-247f5dff70dc-kube-api-access-l9htq\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:15.911334 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.911334 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910838 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.911334 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.910915 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-trusted-ca\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.911334 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.911001 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.911334 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.911136 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:16.411119859 +0000 UTC m=+32.204558789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:15.911572 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.911491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.911900 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.911874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.911978 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.911936 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.912153 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.912287 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zf7\" (UniqueName: \"kubernetes.io/projected/7d1d0d87-bbca-4424-bfad-d0b312732700-kube-api-access-s2zf7\") pod \"network-check-source-6f78d59f99-4nxqr\" (UID: \"7d1d0d87-bbca-4424-bfad-d0b312732700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" Nov 03 13:04:15.912356 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccn5\" (UniqueName: \"kubernetes.io/projected/ec0b3420-306f-4a10-95ff-0bcf4a4f1730-kube-api-access-hccn5\") pod \"volume-data-source-validator-58c9f5f46d-x6jrs\" (UID: \"ec0b3420-306f-4a10-95ff-0bcf4a4f1730\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" Nov 03 13:04:15.912356 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-stats-auth\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.912464 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912364 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-snapshots\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.912464 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjwcb\" (UniqueName: \"kubernetes.io/projected/53c4e88e-1203-4083-98e9-9a2202a3a704-kube-api-access-kjwcb\") pod \"kas-connection-checker-685895c7c9-92c7q\" (UID: \"53c4e88e-1203-4083-98e9-9a2202a3a704\") " pod="kube-system/kas-connection-checker-685895c7c9-92c7q" Nov 03 13:04:15.912464 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912425 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.912464 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71f66428-6b35-45cc-ac46-ce5c07aa958c-config\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.912657 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912476 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99ae820d-6217-425b-9451-247f5dff70dc-config-volume\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:15.912657 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:15.912657 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71f66428-6b35-45cc-ac46-ce5c07aa958c-serving-cert\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.912657 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rnmt\" (UniqueName: \"kubernetes.io/projected/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-kube-api-access-4rnmt\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.912657 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:15.912657 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.912657 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.913004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszmj\" (UniqueName: \"kubernetes.io/projected/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-kube-api-access-gszmj\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.913004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912692 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.913004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-serving-cert\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.913004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912747 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.913004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.913004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912808 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.913004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.913004 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.912853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfgn\" (UniqueName: \"kubernetes.io/projected/05516dd7-e72f-4b64-9707-e401c7e6198b-kube-api-access-cpfgn\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.913428 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.913386 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:15.913542 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.913448 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:16.413434404 +0000 UTC m=+32.206873315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:15.914088 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.914058 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:15.914206 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.914194 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:15.914337 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.914327 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:16.414311622 +0000 UTC m=+32.207750553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:15.915233 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.914357 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/541aac48-8aa9-40e2-88ec-17687fdd5167-cert\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:15.915357 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.914712 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.915445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.915168 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71f66428-6b35-45cc-ac46-ce5c07aa958c-config\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.915538 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.915196 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-config\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.915615 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.915298 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:15.915698 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.915360 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-snapshots\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.915781 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.915450 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50d1326a-e89d-4e45-90c6-9a254a157e48-serving-cert\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.915866 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.915603 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.915943 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.915702 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.916003 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.915774 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:15.916106 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.915884 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.916181 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.916017 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.916234 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.915872 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:15.916283 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.916251 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.916283 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.916256 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:16.416240753 +0000 UTC m=+32.209679682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:15.916462 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:15.916440 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:16.416421796 +0000 UTC m=+32.209860718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:15.916565 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.916470 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.916649 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.916631 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f70a3801-ffa0-4049-aeee-0d886ccb30b3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.916916 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.916900 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-stats-auth\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.916990 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.916971 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-default-certificate\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.917813 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.917786 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71f66428-6b35-45cc-ac46-ce5c07aa958c-serving-cert\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.917885 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.917847 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-serving-cert\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.968642 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.968608 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.970347 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.969789 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/f70a3801-ffa0-4049-aeee-0d886ccb30b3-kube-api-access-dl2z6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:15.970347 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.970288 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2289v\" (UniqueName: \"kubernetes.io/projected/6365006c-9e6f-406f-9102-e3b0b452675d-kube-api-access-2289v\") pod \"kas-connection-checker-685895c7c9-w6sm7\" (UID: \"6365006c-9e6f-406f-9102-e3b0b452675d\") " pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" Nov 03 13:04:15.970347 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.970316 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79g67\" (UniqueName: \"kubernetes.io/projected/50d1326a-e89d-4e45-90c6-9a254a157e48-kube-api-access-79g67\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:15.970781 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.970754 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.971357 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.971330 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rnmt\" (UniqueName: \"kubernetes.io/projected/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-kube-api-access-4rnmt\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:15.972325 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.971970 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzwn\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:15.972898 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.972845 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjwcb\" (UniqueName: \"kubernetes.io/projected/53c4e88e-1203-4083-98e9-9a2202a3a704-kube-api-access-kjwcb\") pod \"kas-connection-checker-685895c7c9-92c7q\" (UID: \"53c4e88e-1203-4083-98e9-9a2202a3a704\") " pod="kube-system/kas-connection-checker-685895c7c9-92c7q" Nov 03 13:04:15.973412 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.973370 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfz2t\" (UniqueName: \"kubernetes.io/projected/541aac48-8aa9-40e2-88ec-17687fdd5167-kube-api-access-rfz2t\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:15.974318 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.973840 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszmj\" (UniqueName: \"kubernetes.io/projected/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-kube-api-access-gszmj\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:15.974318 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.974193 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfds\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:15.974318 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.974276 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qc5\" (UniqueName: \"kubernetes.io/projected/71f66428-6b35-45cc-ac46-ce5c07aa958c-kube-api-access-67qc5\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:15.974563 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.974458 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zf7\" (UniqueName: \"kubernetes.io/projected/7d1d0d87-bbca-4424-bfad-d0b312732700-kube-api-access-s2zf7\") pod \"network-check-source-6f78d59f99-4nxqr\" (UID: \"7d1d0d87-bbca-4424-bfad-d0b312732700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" Nov 03 13:04:15.975203 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.975180 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfgn\" (UniqueName: \"kubernetes.io/projected/05516dd7-e72f-4b64-9707-e401c7e6198b-kube-api-access-cpfgn\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:15.975298 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.975226 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccn5\" (UniqueName: \"kubernetes.io/projected/ec0b3420-306f-4a10-95ff-0bcf4a4f1730-kube-api-access-hccn5\") pod \"volume-data-source-validator-58c9f5f46d-x6jrs\" (UID: \"ec0b3420-306f-4a10-95ff-0bcf4a4f1730\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" Nov 03 13:04:15.975965 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.975949 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q822s\" (UniqueName: \"kubernetes.io/projected/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-kube-api-access-q822s\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:15.976445 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.976418 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trds9\" (UniqueName: \"kubernetes.io/projected/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70-kube-api-access-trds9\") pod \"kas-connection-checker-685895c7c9-f6bgd\" (UID: \"78248e2d-8c0b-47fb-b55b-07e0ffdc1c70\") " pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" Nov 03 13:04:15.996315 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:15.996296 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-92c7q" Nov 03 13:04:16.010119 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.010100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" Nov 03 13:04:16.013908 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.013893 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae820d-6217-425b-9451-247f5dff70dc-tmp-dir\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:16.013968 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.013938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9htq\" (UniqueName: \"kubernetes.io/projected/99ae820d-6217-425b-9451-247f5dff70dc-kube-api-access-l9htq\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:16.014151 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.014128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99ae820d-6217-425b-9451-247f5dff70dc-config-volume\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:16.014277 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.014261 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae820d-6217-425b-9451-247f5dff70dc-tmp-dir\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:16.014341 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.014278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:16.014392 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.014362 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:16.014451 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.014432 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:16.514405647 +0000 UTC m=+32.307844574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:16.014574 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.014557 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99ae820d-6217-425b-9451-247f5dff70dc-config-volume\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:16.017061 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.017041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:16.021772 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.021757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:16.034439 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.034405 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:16.069344 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.064834 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:16.069756 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.069632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" Nov 03 13:04:16.083256 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.077486 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9htq\" (UniqueName: \"kubernetes.io/projected/99ae820d-6217-425b-9451-247f5dff70dc-kube-api-access-l9htq\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:16.083394 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.083273 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:16.259301 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.259267 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" Nov 03 13:04:16.268987 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.268968 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" Nov 03 13:04:16.379318 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.379279 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0b3420_306f_4a10_95ff_0bcf4a4f1730.slice/crio-19202b440d5351427cbaba3d74b7f2e3028bc94d1ad348b0d34a0f4ec9a30da5 WatchSource:0}: Error finding container 19202b440d5351427cbaba3d74b7f2e3028bc94d1ad348b0d34a0f4ec9a30da5: Status 404 returned error can't find the container with id 19202b440d5351427cbaba3d74b7f2e3028bc94d1ad348b0d34a0f4ec9a30da5 Nov 03 13:04:16.379615 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.379590 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs"] Nov 03 13:04:16.388659 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.388376 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-92c7q"] Nov 03 13:04:16.412321 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.412297 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w"] Nov 03 13:04:16.412893 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.412868 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70a3801_ffa0_4049_aeee_0d886ccb30b3.slice/crio-99f4aca45c64a74d0f9897e4f4f028ef82c681cd3d5b7c1bf07a0ceca20ba127 WatchSource:0}: Error finding container 99f4aca45c64a74d0f9897e4f4f028ef82c681cd3d5b7c1bf07a0ceca20ba127: Status 404 returned error can't find the container with id 99f4aca45c64a74d0f9897e4f4f028ef82c681cd3d5b7c1bf07a0ceca20ba127 Nov 03 13:04:16.417307 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.417242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:16.417365 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.417342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:16.417413 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.417388 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:16.417461 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.417421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:16.417560 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.417456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:16.417560 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.417496 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:16.417560 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.417539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:16.417750 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.417687 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:16.417750 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.417701 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:16.417845 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.417752 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:17.417734582 +0000 UTC m=+33.211173508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:16.417845 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.417828 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:17.417816728 +0000 UTC m=+33.211255655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:16.418059 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.417893 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:16.418059 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.417905 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:16.418059 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.417955 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:17.417943711 +0000 UTC m=+33.211382637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:16.418059 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.418022 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:16.418059 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.418059 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:17.418047775 +0000 UTC m=+33.211486701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:16.418325 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.418138 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:16.418325 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.418184 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:17.418161824 +0000 UTC m=+33.211600750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:16.418325 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.418246 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:16.418325 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.418277 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:17.418267415 +0000 UTC m=+33.211706340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:16.418478 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.418333 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:16.418478 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.418366 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:17.418356029 +0000 UTC m=+33.211794956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:16.418882 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.418865 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2pfz7"] Nov 03 13:04:16.419900 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.419880 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d1326a_e89d_4e45_90c6_9a254a157e48.slice/crio-6446669ac9b0972e5f31a23ca710736a72b640a9d9e0d93475d3beef4a08496f WatchSource:0}: Error finding container 6446669ac9b0972e5f31a23ca710736a72b640a9d9e0d93475d3beef4a08496f: Status 404 returned error can't find the container with id 6446669ac9b0972e5f31a23ca710736a72b640a9d9e0d93475d3beef4a08496f Nov 03 13:04:16.447191 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.447163 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk"] Nov 03 13:04:16.447790 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.447740 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f66428_6b35_45cc_ac46_ce5c07aa958c.slice/crio-33c50d8bc8a219dcd53899ebd8d38b978b783083de9739e94bde74167b801ff2 WatchSource:0}: Error finding container 33c50d8bc8a219dcd53899ebd8d38b978b783083de9739e94bde74167b801ff2: Status 404 returned error can't find the container with id 33c50d8bc8a219dcd53899ebd8d38b978b783083de9739e94bde74167b801ff2 Nov 03 13:04:16.488951 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.488923 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1d0d87_bbca_4424_bfad_d0b312732700.slice/crio-377e7a9511a7951a0a720ce9c9f433697c003679361aecdcca5ba678f586af84 WatchSource:0}: Error finding container 377e7a9511a7951a0a720ce9c9f433697c003679361aecdcca5ba678f586af84: Status 404 returned error can't find the container with id 377e7a9511a7951a0a720ce9c9f433697c003679361aecdcca5ba678f586af84 Nov 03 13:04:16.489680 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.489576 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr"] Nov 03 13:04:16.492213 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.492179 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b2bd33_97be_42ee_b7d4_2b86f1de3d7d.slice/crio-8d49330c440a72efe7f3c3950b2c0ccd385e74f9b274e4e4011b884969259e9b WatchSource:0}: Error finding container 8d49330c440a72efe7f3c3950b2c0ccd385e74f9b274e4e4011b884969259e9b: Status 404 returned error can't find the container with id 8d49330c440a72efe7f3c3950b2c0ccd385e74f9b274e4e4011b884969259e9b Nov 03 13:04:16.497251 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.494281 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xhtjf"] Nov 03 13:04:16.506241 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.506215 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541aac48_8aa9_40e2_88ec_17687fdd5167.slice/crio-909314a560bfc46fb1d79e8ec5dc77c4f44c3546b0243b435193b8bd0a285f22 WatchSource:0}: Error finding container 909314a560bfc46fb1d79e8ec5dc77c4f44c3546b0243b435193b8bd0a285f22: Status 404 returned error can't find the container with id 909314a560bfc46fb1d79e8ec5dc77c4f44c3546b0243b435193b8bd0a285f22 Nov 03 13:04:16.506883 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.506836 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hp7ng"] Nov 03 13:04:16.518948 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.518924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:16.519112 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.519096 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:16.519164 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:16.519153 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:17.519137966 +0000 UTC m=+33.312576878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:16.627254 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.627230 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78248e2d_8c0b_47fb_b55b_07e0ffdc1c70.slice/crio-8448ca709f038b2765a1970363c93fe64b5901421a1d46858280ed444f73d7cb WatchSource:0}: Error finding container 8448ca709f038b2765a1970363c93fe64b5901421a1d46858280ed444f73d7cb: Status 404 returned error can't find the container with id 8448ca709f038b2765a1970363c93fe64b5901421a1d46858280ed444f73d7cb Nov 03 13:04:16.628345 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.628256 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f6bgd"] Nov 03 13:04:16.628531 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:16.628500 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6365006c_9e6f_406f_9102_e3b0b452675d.slice/crio-d8b8a280f1ea9d9f11e039953b6bf403ecdb5f34fc62d4493024756e8f9c208d WatchSource:0}: Error finding container d8b8a280f1ea9d9f11e039953b6bf403ecdb5f34fc62d4493024756e8f9c208d: Status 404 returned error can't find the container with id d8b8a280f1ea9d9f11e039953b6bf403ecdb5f34fc62d4493024756e8f9c208d Nov 03 13:04:16.629593 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.629571 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w6sm7"] Nov 03 13:04:16.629785 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.629765 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:16.683012 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.682980 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:04:16.683461 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.682986 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnhgg\"" Nov 03 13:04:16.820328 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.820294 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" event={"ID":"78248e2d-8c0b-47fb-b55b-07e0ffdc1c70","Type":"ContainerStarted","Data":"1ac10a064f283fbaa6555bc510503213996236366df65416b80397549c4b1c3f"} Nov 03 13:04:16.820510 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.820342 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" event={"ID":"78248e2d-8c0b-47fb-b55b-07e0ffdc1c70","Type":"ContainerStarted","Data":"8448ca709f038b2765a1970363c93fe64b5901421a1d46858280ed444f73d7cb"} Nov 03 13:04:16.821513 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.821487 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" event={"ID":"7d1d0d87-bbca-4424-bfad-d0b312732700","Type":"ContainerStarted","Data":"377e7a9511a7951a0a720ce9c9f433697c003679361aecdcca5ba678f586af84"} Nov 03 13:04:16.822642 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.822622 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" event={"ID":"f70a3801-ffa0-4049-aeee-0d886ccb30b3","Type":"ContainerStarted","Data":"99f4aca45c64a74d0f9897e4f4f028ef82c681cd3d5b7c1bf07a0ceca20ba127"} Nov 03 13:04:16.823729 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.823712 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" event={"ID":"71f66428-6b35-45cc-ac46-ce5c07aa958c","Type":"ContainerStarted","Data":"33c50d8bc8a219dcd53899ebd8d38b978b783083de9739e94bde74167b801ff2"} Nov 03 13:04:16.824821 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.824803 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" event={"ID":"50d1326a-e89d-4e45-90c6-9a254a157e48","Type":"ContainerStarted","Data":"6446669ac9b0972e5f31a23ca710736a72b640a9d9e0d93475d3beef4a08496f"} Nov 03 13:04:16.825840 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.825821 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" event={"ID":"ec0b3420-306f-4a10-95ff-0bcf4a4f1730","Type":"ContainerStarted","Data":"19202b440d5351427cbaba3d74b7f2e3028bc94d1ad348b0d34a0f4ec9a30da5"} Nov 03 13:04:16.827165 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.827148 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-92c7q" event={"ID":"53c4e88e-1203-4083-98e9-9a2202a3a704","Type":"ContainerStarted","Data":"b5a72de88ef91a933889f6efc17ee4591a113c958a7c84d9f6d9e67ff6872305"} Nov 03 13:04:16.827248 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.827177 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-92c7q" event={"ID":"53c4e88e-1203-4083-98e9-9a2202a3a704","Type":"ContainerStarted","Data":"3d8e2658eb5bb2e05bd690d7010519ef355254e88c0cd998348c646617a744bb"} Nov 03 13:04:16.828247 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.828224 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xhtjf" event={"ID":"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d","Type":"ContainerStarted","Data":"8d49330c440a72efe7f3c3950b2c0ccd385e74f9b274e4e4011b884969259e9b"} Nov 03 13:04:16.829537 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.829515 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" event={"ID":"6365006c-9e6f-406f-9102-e3b0b452675d","Type":"ContainerStarted","Data":"93f9db55daa673b9d593d33b3f78b7d87943959fb597d13933486fc61313605e"} Nov 03 13:04:16.829626 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.829554 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" event={"ID":"6365006c-9e6f-406f-9102-e3b0b452675d","Type":"ContainerStarted","Data":"d8b8a280f1ea9d9f11e039953b6bf403ecdb5f34fc62d4493024756e8f9c208d"} Nov 03 13:04:16.830676 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:16.830658 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hp7ng" event={"ID":"541aac48-8aa9-40e2-88ec-17687fdd5167","Type":"ContainerStarted","Data":"909314a560bfc46fb1d79e8ec5dc77c4f44c3546b0243b435193b8bd0a285f22"} Nov 03 13:04:17.426248 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.426212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:17.426443 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.426261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:17.426443 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.426322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:17.426443 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.426345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:17.426443 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.426372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:17.426443 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426388 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:17.426443 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426410 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426467 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426394 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.426486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426486 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426475 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.426457052 +0000 UTC m=+35.219895979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426538 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426551 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426558 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.426545839 +0000 UTC m=+35.219984758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426579 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.426567153 +0000 UTC m=+35.220006072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426539 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426597 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.426588856 +0000 UTC m=+35.220027775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426616 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.426607711 +0000 UTC m=+35.220046631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426634 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.42662716 +0000 UTC m=+35.220066073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:17.426801 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.426701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:17.427271 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.426834 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.426822541 +0000 UTC m=+35.220261453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:17.527587 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.527553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:17.527764 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.527718 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:17.527810 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:17.527797 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.527776003 +0000 UTC m=+35.321214917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:17.799398 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.799288 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" podStartSLOduration=453.799272681 podStartE2EDuration="7m33.799272681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:17.798021131 +0000 UTC m=+33.591460069" watchObservedRunningTime="2026-11-03 13:04:17.799272681 +0000 UTC m=+33.592711615" Nov 03 13:04:17.912058 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:17.911996 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-92c7q" podStartSLOduration=453.911978523 podStartE2EDuration="7m33.911978523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:17.909997676 +0000 UTC m=+33.703436607" watchObservedRunningTime="2026-11-03 13:04:17.911978523 +0000 UTC m=+33.705417460" Nov 03 13:04:18.538045 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.537951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:18.538308 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.538060 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:18.538757 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:18.538732 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:18.538868 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:18.538805 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:04:50.538786148 +0000 UTC m=+66.332225061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : secret "metrics-daemon-secret" not found Nov 03 13:04:18.551495 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.551466 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:18.639515 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.639481 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:18.647895 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.647834 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:18.747453 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.747231 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:18.760956 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.760857 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:18.930442 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.930362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:18.944750 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:18.944476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:19.170932 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.169720 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" podStartSLOduration=455.169700102 podStartE2EDuration="7m35.169700102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:18.024598934 +0000 UTC m=+33.818037866" watchObservedRunningTime="2026-11-03 13:04:19.169700102 +0000 UTC m=+34.963139033" Nov 03 13:04:19.170932 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.170652 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6ct64"] Nov 03 13:04:19.173708 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.173682 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:19.180486 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:19.180454 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed1b9f7_3769_46a0_b8ba_63eb8d17a952.slice/crio-2d2fe075bfbd3e9985a88fbc3db540d5726c9f1e6881e7793b33bf4d34850e4e WatchSource:0}: Error finding container 2d2fe075bfbd3e9985a88fbc3db540d5726c9f1e6881e7793b33bf4d34850e4e: Status 404 returned error can't find the container with id 2d2fe075bfbd3e9985a88fbc3db540d5726c9f1e6881e7793b33bf4d34850e4e Nov 03 13:04:19.338064 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.338033 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nh6c9"] Nov 03 13:04:19.340818 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:19.340773 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1651b048_50ba_4391_8382_60f9cffd1fa7.slice/crio-35f342274cdc3da99c9116e872a17dae775a0217c13fe74d46312f367b1bf550 WatchSource:0}: Error finding container 35f342274cdc3da99c9116e872a17dae775a0217c13fe74d46312f367b1bf550: Status 404 returned error can't find the container with id 35f342274cdc3da99c9116e872a17dae775a0217c13fe74d46312f367b1bf550 Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.448486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.448555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.448593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.448633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.448677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.448726 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.448792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.448933 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.448949 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449012 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.448992324 +0000 UTC m=+39.242431261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449520 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449580 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.449563605 +0000 UTC m=+39.243002540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449639 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449684 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.44966222 +0000 UTC m=+39.243101147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:19.450134 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449742 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:19.451049 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449775 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.44976526 +0000 UTC m=+39.243204193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:19.451049 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449823 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:19.451049 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449865 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.449856196 +0000 UTC m=+39.243295122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:19.451049 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449922 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:19.451049 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449932 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:19.451049 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.449963 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.449953273 +0000 UTC m=+39.243392198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:19.451049 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.450009 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.450003905 +0000 UTC m=+39.243442830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:19.550645 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.550251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:19.550645 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.550522 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:19.550645 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:19.550584 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.550567127 +0000 UTC m=+39.344006052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:19.844119 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.843911 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nh6c9" event={"ID":"1651b048-50ba-4391-8382-60f9cffd1fa7","Type":"ContainerStarted","Data":"35f342274cdc3da99c9116e872a17dae775a0217c13fe74d46312f367b1bf550"} Nov 03 13:04:19.846209 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.846089 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6ct64" event={"ID":"eed1b9f7-3769-46a0-b8ba-63eb8d17a952","Type":"ContainerStarted","Data":"2d2fe075bfbd3e9985a88fbc3db540d5726c9f1e6881e7793b33bf4d34850e4e"} Nov 03 13:04:19.850950 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:19.850185 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" event={"ID":"ec0b3420-306f-4a10-95ff-0bcf4a4f1730","Type":"ContainerStarted","Data":"313b48c0edf3a1ca0c48d18ccd3befce5976fd6661672ac24f81d8befb4d3f44"} Nov 03 13:04:20.051893 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:20.051818 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:21.301536 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:21.301489 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:21.413205 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:21.413171 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:22.864510 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.864474 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" event={"ID":"71f66428-6b35-45cc-ac46-ce5c07aa958c","Type":"ContainerStarted","Data":"6d1ceb0463223c01ba86329bf485d3fcc60b314381623d4f3fe5dc28020ea909"} Nov 03 13:04:22.869775 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.869604 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/0.log" Nov 03 13:04:22.869775 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.869647 2582 generic.go:356] "Generic (PLEG): container finished" podID="50d1326a-e89d-4e45-90c6-9a254a157e48" containerID="57c1230f107cceed6e6db3639774d2360eb867210693b4789852f53d0e493490" exitCode=255 Nov 03 13:04:22.869934 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.869748 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" event={"ID":"50d1326a-e89d-4e45-90c6-9a254a157e48","Type":"ContainerDied","Data":"57c1230f107cceed6e6db3639774d2360eb867210693b4789852f53d0e493490"} Nov 03 13:04:22.871024 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.870110 2582 scope.go:122] "RemoveContainer" containerID="57c1230f107cceed6e6db3639774d2360eb867210693b4789852f53d0e493490" Nov 03 13:04:22.876687 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.876656 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xhtjf" event={"ID":"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d","Type":"ContainerStarted","Data":"8a2449557043cfc2f0b4374edd44964b1e13ac7f57a9e24cb784c08f47791bfd"} Nov 03 13:04:22.881090 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.880475 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" event={"ID":"f70a3801-ffa0-4049-aeee-0d886ccb30b3","Type":"ContainerStarted","Data":"27781848cbbbbfa98733d1cb8339cb56c64b7c45ff8a23a0958d93c05e56d05e"} Nov 03 13:04:22.980043 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.979975 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" podStartSLOduration=427.677753588 podStartE2EDuration="7m12.979954291s" totalImagesPullingTime="5.302200703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:22.979255973 +0000 UTC m=+38.772694909" watchObservedRunningTime="2026-11-03 13:04:22.979954291 +0000 UTC m=+38.773393227" Nov 03 13:04:22.980530 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:22.980479 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" podStartSLOduration=430.624766659 podStartE2EDuration="7m12.98046521s" totalImagesPullingTime="2.355698551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:19.96887251 +0000 UTC m=+35.762311445" watchObservedRunningTime="2026-11-03 13:04:22.98046521 +0000 UTC m=+38.773904146" Nov 03 13:04:23.097689 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.096718 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-xhtjf" podStartSLOduration=427.855918704 podStartE2EDuration="7m13.096700316s" totalImagesPullingTime="5.240781612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:23.095340373 +0000 UTC m=+38.888779308" watchObservedRunningTime="2026-11-03 13:04:23.096700316 +0000 UTC m=+38.890139251" Nov 03 13:04:23.176691 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.176656 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cq7lf"] Nov 03 13:04:23.198221 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.198189 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.210614 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.208863 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" podStartSLOduration=427.619426992 podStartE2EDuration="7m13.208845563s" totalImagesPullingTime="5.589418571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:23.208250747 +0000 UTC m=+39.001689689" watchObservedRunningTime="2026-11-03 13:04:23.208845563 +0000 UTC m=+39.002284499" Nov 03 13:04:23.218278 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.218016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:23.232986 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.232948 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cq7lf"] Nov 03 13:04:23.263463 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.263430 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 13:04:23.263624 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.263515 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 13:04:23.263782 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.263762 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pvzn\"" Nov 03 13:04:23.298814 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.298681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7h6\" (UniqueName: \"kubernetes.io/projected/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-api-access-jw7h6\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.298814 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.298734 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8425f3a6-d496-41eb-bb83-c65f6011890b-data-volume\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.299050 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.298964 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.299050 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.299010 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8425f3a6-d496-41eb-bb83-c65f6011890b-crio-socket\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.299209 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.299178 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.377774 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.377683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:23.401253 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.400575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8425f3a6-d496-41eb-bb83-c65f6011890b-data-volume\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.401253 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.400727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.401253 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.400763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8425f3a6-d496-41eb-bb83-c65f6011890b-crio-socket\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.401253 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.400834 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.401253 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.400879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7h6\" (UniqueName: \"kubernetes.io/projected/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-api-access-jw7h6\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.401662 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.401540 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8425f3a6-d496-41eb-bb83-c65f6011890b-data-volume\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.401662 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.401619 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:04:23.401763 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.401680 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.901658884 +0000 UTC m=+39.695097810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : configmap "kube-rbac-proxy" not found Nov 03 13:04:23.401763 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.401740 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8425f3a6-d496-41eb-bb83-c65f6011890b-crio-socket\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.401871 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.401839 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:23.401923 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.401878 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.901865815 +0000 UTC m=+39.695304744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:23.460213 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.460180 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7h6\" (UniqueName: \"kubernetes.io/projected/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-api-access-jw7h6\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.496299 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.496228 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:23.501506 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.501304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:23.501506 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.501378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:23.501506 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.501409 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:23.501506 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.501438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:23.501506 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501462 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.501443702 +0000 UTC m=+47.294882614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:23.501506 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.501503 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501512 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.501532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501553 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.501543361 +0000 UTC m=+47.294982286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.501584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501590 2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501622 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.50161134 +0000 UTC m=+47.295050287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501659 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501667 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501671 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501693 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.501686012 +0000 UTC m=+47.295124956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501704 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.501698392 +0000 UTC m=+47.295137304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501737 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501741 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501744 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501762 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.501755036 +0000 UTC m=+47.295193994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:23.501860 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.501772 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.501766769 +0000 UTC m=+47.295205681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:23.603093 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.602897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:23.603093 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.603011 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:23.603258 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.603102 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.603064273 +0000 UTC m=+47.396503201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:23.612980 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.612945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:23.885892 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.885857 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hp7ng" event={"ID":"541aac48-8aa9-40e2-88ec-17687fdd5167","Type":"ContainerStarted","Data":"28bb07429b261b864968b4fb83030a9a2b6a9b3a76fe8d18df57e36d10c20d09"} Nov 03 13:04:23.887846 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.887813 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" event={"ID":"7d1d0d87-bbca-4424-bfad-d0b312732700","Type":"ContainerStarted","Data":"3fc0ec0d6892f9a2b46284fbb657abe92b52021a86e99d9ccf0810685694dd0e"} Nov 03 13:04:23.889897 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.889542 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/1.log" Nov 03 13:04:23.889990 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.889970 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/0.log" Nov 03 13:04:23.890131 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.890003 2582 generic.go:356] "Generic (PLEG): container finished" podID="50d1326a-e89d-4e45-90c6-9a254a157e48" containerID="17d482226f7b4a1122e534d47d3499d4d826d795c3594799547107c9df888788" exitCode=255 Nov 03 13:04:23.890131 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.890065 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" event={"ID":"50d1326a-e89d-4e45-90c6-9a254a157e48","Type":"ContainerDied","Data":"17d482226f7b4a1122e534d47d3499d4d826d795c3594799547107c9df888788"} Nov 03 13:04:23.890219 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.890137 2582 scope.go:122] "RemoveContainer" containerID="57c1230f107cceed6e6db3639774d2360eb867210693b4789852f53d0e493490" Nov 03 13:04:23.890332 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.890314 2582 scope.go:122] "RemoveContainer" containerID="17d482226f7b4a1122e534d47d3499d4d826d795c3594799547107c9df888788" Nov 03 13:04:23.890549 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.890524 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2pfz7_openshift-console-operator(50d1326a-e89d-4e45-90c6-9a254a157e48)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" podUID="50d1326a-e89d-4e45-90c6-9a254a157e48" Nov 03 13:04:23.905229 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.905198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.905362 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.905347 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:23.905409 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:23.905385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:23.905455 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.905412 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:24.905390942 +0000 UTC m=+40.698829870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:23.905455 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.905446 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:04:23.905539 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:23.905504 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:24.90548768 +0000 UTC m=+40.698926592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : configmap "kube-rbac-proxy" not found Nov 03 13:04:24.000687 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.000631 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hp7ng" podStartSLOduration=2.691471572 podStartE2EDuration="9.000617018s" totalImagesPullingTime="6.309145446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:23.999218942 +0000 UTC m=+39.792657876" watchObservedRunningTime="2026-11-03 13:04:24.000617018 +0000 UTC m=+39.794055997" Nov 03 13:04:24.114032 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.113928 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" podStartSLOduration=425.101822 podStartE2EDuration="7m12.113914077s" totalImagesPullingTime="7.012092077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:24.112494739 +0000 UTC m=+39.905933673" watchObservedRunningTime="2026-11-03 13:04:24.113914077 +0000 UTC m=+39.907353011" Nov 03 13:04:24.146045 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.145997 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:24.279006 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.278969 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:24.370115 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.370037 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g"] Nov 03 13:04:24.385613 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.385586 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" Nov 03 13:04:24.427715 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.427680 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g"] Nov 03 13:04:24.504134 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.504094 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:24.507783 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.507759 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:24.507938 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.507818 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 13:04:24.512459 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.512432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nc4l\" (UniqueName: \"kubernetes.io/projected/cc64bc02-50f7-4594-8eaa-3a4ec97763f1-kube-api-access-6nc4l\") pod \"migrator-66df5d496b-82p5g\" (UID: \"cc64bc02-50f7-4594-8eaa-3a4ec97763f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" Nov 03 13:04:24.529927 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.529902 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8p2l7\"" Nov 03 13:04:24.613620 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.613561 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nc4l\" (UniqueName: \"kubernetes.io/projected/cc64bc02-50f7-4594-8eaa-3a4ec97763f1-kube-api-access-6nc4l\") pod \"migrator-66df5d496b-82p5g\" (UID: \"cc64bc02-50f7-4594-8eaa-3a4ec97763f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" Nov 03 13:04:24.629023 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.628917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:24.671848 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.671811 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nc4l\" (UniqueName: \"kubernetes.io/projected/cc64bc02-50f7-4594-8eaa-3a4ec97763f1-kube-api-access-6nc4l\") pod \"migrator-66df5d496b-82p5g\" (UID: \"cc64bc02-50f7-4594-8eaa-3a4ec97763f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" Nov 03 13:04:24.708703 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.708666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" Nov 03 13:04:24.766634 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.763310 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:24.897010 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.896912 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nh6c9" event={"ID":"1651b048-50ba-4391-8382-60f9cffd1fa7","Type":"ContainerStarted","Data":"503e41536d5f5a2168baadcb5e3a328dc24b8b460048d44b337715b380d9b132"} Nov 03 13:04:24.897010 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.896993 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:24.898626 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.898600 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/1.log" Nov 03 13:04:24.899130 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.899111 2582 scope.go:122] "RemoveContainer" containerID="17d482226f7b4a1122e534d47d3499d4d826d795c3594799547107c9df888788" Nov 03 13:04:24.899311 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:24.899287 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2pfz7_openshift-console-operator(50d1326a-e89d-4e45-90c6-9a254a157e48)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" podUID="50d1326a-e89d-4e45-90c6-9a254a157e48" Nov 03 13:04:24.917643 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.917607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:24.917779 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:24.917710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:24.917779 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:24.917745 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:04:24.917895 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:24.917815 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:26.917799902 +0000 UTC m=+42.711238814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : configmap "kube-rbac-proxy" not found Nov 03 13:04:24.917895 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:24.917856 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:24.918012 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:24.917909 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:26.917897756 +0000 UTC m=+42.711336668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:25.013002 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:25.012952 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nh6c9" podStartSLOduration=35.560373262 podStartE2EDuration="40.012938699s" totalImagesPullingTime="4.452565437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:25.012344261 +0000 UTC m=+40.805783195" watchObservedRunningTime="2026-11-03 13:04:25.012938699 +0000 UTC m=+40.806377632" Nov 03 13:04:25.090538 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:25.090475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:25.116219 ip-10-0-129-63 kubenswrapper[2582]: W1103 13:04:25.116177 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc64bc02_50f7_4594_8eaa_3a4ec97763f1.slice/crio-3148eac97ea1103d424f88bf6b628e4061236c7ab2a3a51cb556742cb08562cf WatchSource:0}: Error finding container 3148eac97ea1103d424f88bf6b628e4061236c7ab2a3a51cb556742cb08562cf: Status 404 returned error can't find the container with id 3148eac97ea1103d424f88bf6b628e4061236c7ab2a3a51cb556742cb08562cf Nov 03 13:04:25.117218 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:25.117195 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g"] Nov 03 13:04:25.903141 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:25.903103 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" event={"ID":"cc64bc02-50f7-4594-8eaa-3a4ec97763f1","Type":"ContainerStarted","Data":"3148eac97ea1103d424f88bf6b628e4061236c7ab2a3a51cb556742cb08562cf"} Nov 03 13:04:26.018247 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:26.018214 2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:26.018377 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:26.018272 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:26.018632 ip-10-0-129-63 kubenswrapper[2582]: I1103 13:04:26.018620 2582 scope.go:122] "RemoveContainer" containerID="17d482226f7b4a1122e534d47d3499d4d826d795c3594799547107c9df888788" Nov 03 13:04:26.018836 ip-10-0-129-63 kubenswrapper[2582]: E1103 13:04:26.018818 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2pfz7_openshift-console-operator(50d1326a-e89d-4e45-90c6-9a254a157e48)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" podUID="50d1326a-e89d-4e45-90c6-9a254a157e48" Nov 03 13:04:26.721141 ip-10-0-129-63 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 13:04:26.734503 ip-10-0-129-63 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 13:04:26.734676 ip-10-0-129-63 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 13:04:26.734729 ip-10-0-129-63 systemd[1]: kubelet.service: Consumed 3.134s CPU time. Nov 03 13:04:26.744655 ip-10-0-129-63 systemd[1]: Starting Kubernetes Kubelet... Nov 03 13:04:26.842049 ip-10-0-129-63 kubenswrapper[4924]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:26.842049 ip-10-0-129-63 kubenswrapper[4924]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 13:04:26.842049 ip-10-0-129-63 kubenswrapper[4924]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:26.842049 ip-10-0-129-63 kubenswrapper[4924]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:26.844454 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844356 4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:26.844454 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844453 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:26.844454 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844457 4924 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:26.844454 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844460 4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844463 4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844466 4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844469 4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844472 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844474 4924 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844477 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844480 4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844482 4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844485 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844488 4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844491 4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844502 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844505 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844508 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844511 4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844514 4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844516 4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844519 4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:26.844660 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844522 4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844526 4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844530 4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844535 4924 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844540 4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844545 4924 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844549 4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844552 4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844556 4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844559 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844562 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844565 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844568 4924 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844570 4924 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844574 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844579 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844583 4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844587 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844591 4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:26.845287 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844594 4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844598 4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844602 4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844606 4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844610 4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844615 4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844618 4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844621 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844624 4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844629 4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844634 4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844638 4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844641 4924 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844643 4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844647 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844651 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844653 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844664 4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844667 4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:26.845930 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844670 4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844673 4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844677 4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844681 4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844686 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844690 4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844695 4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844700 4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844704 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844708 4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844712 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844714 4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844717 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844721 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844725 4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844729 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844733 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844738 4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844742 4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844746 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:26.846596 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844750 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844755 4924 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844760 4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844764 4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844767 4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844796 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844801 4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844808 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844811 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844814 4924 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844818 4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844822 4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844826 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844831 4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844836 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844839 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844841 4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844844 4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844847 4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844850 4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:26.847221 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844852 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844855 4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844857 4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844860 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844862 4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.844865 4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845898 4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845907 4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845913 4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845917 4924 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845923 4924 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845927 4924 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845932 4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845937 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845943 4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845951 4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845956 4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845959 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845962 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845967 4924 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:26.847782 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845970 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845973 4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845976 4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845978 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845981 4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845984 4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845988 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845992 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.845997 4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846001 4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846007 4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846011 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846015 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846019 4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846023 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846027 4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846031 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846035 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846039 4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846043 4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:26.848400 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846049 4924 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846052 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846054 4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846057 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846060 4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846063 4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846080 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846099 4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846103 4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846107 4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846111 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846116 4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846128 4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846131 4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846133 4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846136 4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846139 4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846141 4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846146 4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:26.849028 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846150 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846154 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846158 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846163 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846167 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846172 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846175 4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846178 4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846180 4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846183 4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846185 4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846188 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846191 4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846193 4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846196 4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846199 4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846201 4924 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846204 4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846207 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846210 4924 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:26.849613 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846212 4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846221 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846224 4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846226 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846231 4924 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846234 4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846237 4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846241 4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846245 4924 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846249 4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846254 4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846257 4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846259 4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846262 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846265 4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846267 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846270 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846274 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846278 4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:26.850222 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846282 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846287 4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846291 4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846295 4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846299 4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846303 4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846308 4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846312 4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846316 4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846320 4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846324 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846330 4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846333 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846336 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846954 4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846965 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846970 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846977 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846981 4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846983 4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:26.850801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846986 4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846989 4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846992 4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846994 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.846997 4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847000 4924 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847002 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847005 4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847008 4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847011 4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847013 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847016 4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847019 4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847022 4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847024 4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847027 4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847030 4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847032 4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847035 4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847038 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:26.851372 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847040 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847043 4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847046 4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847048 4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847051 4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847053 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847056 4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847059 4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847063 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847089 4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847096 4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847102 4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847108 4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847113 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847118 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847122 4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847127 4924 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847130 4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847133 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:26.851984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847136 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847138 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847142 4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847146 4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847150 4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847154 4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847159 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847162 4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847165 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847168 4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847170 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847173 4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847175 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847178 4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847180 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847184 4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847187 4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847189 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847193 4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847196 4924 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:26.852637 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847198 4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847201 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847206 4924 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847209 4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847211 4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847216 4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847220 4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847224 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847229 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847236 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847242 4924 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847246 4924 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847251 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847255 4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847260 4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847264 4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847268 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847272 4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847275 4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:26.853241 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847278 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847281 4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847283 4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847286 4924 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847289 4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847291 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847294 4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847297 4924 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847299 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847302 4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847305 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847308 4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847310 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847313 4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847315 4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847319 4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847322 4924 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847325 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847328 4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:26.853773 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847330 4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847333 4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.847336 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847456 4924 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847476 4924 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847489 4924 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847494 4924 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847498 4924 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847502 4924 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847506 4924 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847511 4924 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847514 4924 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847518 4924 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847524 4924 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847530 4924 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847535 4924 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847540 4924 flags.go:64] FLAG: --cgroup-root="" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847543 4924 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847546 4924 flags.go:64] FLAG: --client-ca-file="" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847549 4924 flags.go:64] FLAG: --cloud-provider="external" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847552 4924 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847560 4924 flags.go:64] FLAG: --cluster-domain="" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847563 4924 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 13:04:26.854453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847566 4924 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847571 4924 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847576 4924 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847584 4924 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847590 4924 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847596 4924 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847609 4924 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847614 4924 flags.go:64] FLAG: --contention-profiling="false" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847620 4924 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847624 4924 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847630 4924 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847635 4924 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847644 4924 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847648 4924 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847651 4924 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847654 4924 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847657 4924 flags.go:64] FLAG: --enable-server="true" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847660 4924 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847669 4924 flags.go:64] FLAG: --event-burst="100" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847674 4924 flags.go:64] FLAG: --event-qps="50" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847678 4924 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847683 4924 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847686 4924 flags.go:64] FLAG: --eviction-hard="" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847692 4924 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 13:04:26.855123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847697 4924 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847702 4924 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847708 4924 flags.go:64] FLAG: --eviction-soft="" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847713 4924 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847718 4924 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847721 4924 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847724 4924 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847728 4924 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847731 4924 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847735 4924 flags.go:64] FLAG: --feature-gates="" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847742 4924 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847747 4924 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847752 4924 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847757 4924 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847762 4924 flags.go:64] FLAG: --healthz-port="10248" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847772 4924 flags.go:64] FLAG: --help="false" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847776 4924 flags.go:64] FLAG: --hostname-override="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847779 4924 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847782 4924 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847785 4924 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847790 4924 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847794 4924 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847797 4924 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 13:04:26.855935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847800 4924 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847803 4924 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847807 4924 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847809 4924 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847813 4924 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847816 4924 flags.go:64] FLAG: --kube-reserved="" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847819 4924 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847822 4924 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847825 4924 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847828 4924 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847831 4924 flags.go:64] FLAG: --lock-file="" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847833 4924 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847837 4924 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847842 4924 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847851 4924 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847856 4924 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847861 4924 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847866 4924 flags.go:64] FLAG: --logging-format="text" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847871 4924 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847876 4924 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847881 4924 flags.go:64] FLAG: --manifest-url="" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847886 4924 flags.go:64] FLAG: --manifest-url-header="" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847893 4924 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847898 4924 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847903 4924 flags.go:64] FLAG: --max-pods="110" Nov 03 13:04:26.856601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847912 4924 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847916 4924 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847919 4924 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847922 4924 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847927 4924 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847930 4924 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847935 4924 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847946 4924 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847950 4924 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847953 4924 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847960 4924 flags.go:64] FLAG: --pod-cidr="" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847965 4924 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847969 4924 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847973 4924 flags.go:64] FLAG: --pods-per-core="0" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847976 4924 flags.go:64] FLAG: --port="10250" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847979 4924 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847984 4924 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-037071a651035b0b9" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847990 4924 flags.go:64] FLAG: --qos-reserved="" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847994 4924 flags.go:64] FLAG: --read-only-port="10255" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.847997 4924 flags.go:64] FLAG: --register-node="true" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848000 4924 flags.go:64] FLAG: --register-with-taints="" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848004 4924 flags.go:64] FLAG: --registry-burst="10" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848007 4924 flags.go:64] FLAG: --registry-qps="5" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848010 4924 flags.go:64] FLAG: --reserved-cpus="" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848013 4924 flags.go:64] FLAG: --reserved-memory="" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848017 4924 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 13:04:26.857341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848020 4924 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848024 4924 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848026 4924 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848031 4924 flags.go:64] FLAG: --runonce="false" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848036 4924 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848041 4924 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848046 4924 flags.go:64] FLAG: --seccomp-default="false" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848057 4924 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848060 4924 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848064 4924 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848080 4924 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848084 4924 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848087 4924 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848090 4924 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848093 4924 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848096 4924 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848101 4924 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848104 4924 flags.go:64] FLAG: --system-cgroups="" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848108 4924 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848117 4924 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848122 4924 flags.go:64] FLAG: --tls-cert-file="" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848125 4924 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848133 4924 flags.go:64] FLAG: --tls-min-version="" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848137 4924 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848142 4924 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 13:04:26.858042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848147 4924 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848152 4924 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848156 4924 flags.go:64] FLAG: --v="2" Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848169 4924 flags.go:64] FLAG: --version="false" Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848177 4924 flags.go:64] FLAG: --vmodule="" Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848183 4924 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848188 4924 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848406 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848413 4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848418 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848422 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848428 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848433 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848438 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848443 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848452 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848455 4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848459 4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848461 4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848464 4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848467 4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:26.858743 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848469 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848472 4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848476 4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848479 4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848481 4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848484 4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848487 4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848490 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848492 4924 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848495 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848498 4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848500 4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848503 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848506 4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848509 4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848512 4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848514 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848517 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848519 4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:26.859344 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848522 4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848526 4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848530 4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848535 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848539 4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848544 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848549 4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848553 4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848565 4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848569 4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848572 4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848574 4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848579 4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848583 4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848587 4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848593 4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848596 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848599 4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848602 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:26.859867 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848606 4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848610 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848615 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848619 4924 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848621 4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848624 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848627 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848630 4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848634 4924 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848638 4924 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848642 4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848647 4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848652 4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848657 4924 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848662 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848667 4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848670 4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848673 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848676 4924 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848678 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:26.860399 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848682 4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848686 4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848703 4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848706 4924 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848709 4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848712 4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848715 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848719 4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848726 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848731 4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848735 4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848740 4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848744 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848751 4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848758 4924 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848763 4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848768 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848772 4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848777 4924 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848781 4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:26.861026 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848786 4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848790 4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848795 4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848799 4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848803 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848808 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848811 4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848814 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848817 4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848819 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848822 4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848825 4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848829 4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848833 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.848879 4924 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:26.861693 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848939 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848944 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848947 4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848951 4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848955 4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848960 4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848966 4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848969 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848971 4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848974 4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848978 4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848982 4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848987 4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848992 4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.848996 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849001 4924 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849006 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849011 4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849015 4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:26.862232 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849018 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849021 4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849024 4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849027 4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849032 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849039 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849044 4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849048 4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849050 4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849053 4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849056 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849058 4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849061 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849064 4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849084 4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849089 4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849093 4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849098 4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849102 4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:26.862786 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849107 4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849109 4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849112 4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849116 4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849120 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849125 4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849129 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849133 4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849138 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849143 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849148 4924 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849152 4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849157 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849160 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849163 4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849167 4924 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849172 4924 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849176 4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849183 4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849186 4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:26.863314 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849189 4924 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849192 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849195 4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849198 4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849201 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849205 4924 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849209 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849214 4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849224 4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849227 4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849231 4924 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849236 4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849243 4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849247 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849252 4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849256 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849261 4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849265 4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849268 4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:26.864032 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849271 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849277 4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849283 4924 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849288 4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849293 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849298 4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849301 4924 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849304 4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849307 4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849309 4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849312 4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849319 4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849322 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849325 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849327 4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849330 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849333 4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849336 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849338 4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:26.864713 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849341 4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849345 4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849350 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849362 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849368 4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849373 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849381 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849389 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849393 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.849398 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.856344 4924 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.856366 4924 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.856396 4924 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.856405 4924 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856502 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856510 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856516 4924 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856521 4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856526 4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:26.865351 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856531 4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856535 4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856540 4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856544 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856549 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856553 4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856558 4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856563 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856567 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856571 4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856576 4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856580 4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856585 4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856589 4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856593 4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856597 4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856602 4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856607 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856612 4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:26.865927 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856616 4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856621 4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856626 4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856631 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856635 4924 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856639 4924 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856644 4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856648 4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856653 4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856657 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856661 4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856665 4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856670 4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856674 4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856679 4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856683 4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856687 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856692 4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856697 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856701 4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:26.866521 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856705 4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856709 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856720 4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856725 4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856730 4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856734 4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856738 4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856743 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856747 4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856751 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856758 4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856764 4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856782 4924 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856787 4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856792 4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856809 4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856814 4924 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856822 4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856829 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:26.867172 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856835 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856840 4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856844 4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856848 4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856853 4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856857 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856862 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856866 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856871 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856876 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856880 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856885 4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856889 4924 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856893 4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856898 4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856903 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856914 4924 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856919 4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856924 4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856928 4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:26.867801 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856932 4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856936 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856941 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856945 4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856949 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856953 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856958 4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856962 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856966 4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856971 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856975 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856981 4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856985 4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856989 4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856993 4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.856997 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857002 4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857006 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857010 4924 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857014 4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:26.868458 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857019 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857024 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857029 4924 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.857082 4924 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857310 4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857317 4924 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857322 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857327 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857331 4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857342 4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857348 4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857352 4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857357 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857361 4924 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857365 4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857369 4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:26.869099 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857374 4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857379 4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857384 4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857388 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857393 4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857397 4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857401 4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857405 4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857410 4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857414 4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857418 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857423 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857427 4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857431 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857436 4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857440 4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857445 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857450 4924 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857454 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:26.869600 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857459 4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857463 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857467 4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857471 4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857476 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857480 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857484 4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857495 4924 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857499 4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857503 4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857507 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857512 4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857516 4924 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857520 4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857524 4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857528 4924 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857533 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857537 4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857541 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:26.870243 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857546 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857550 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857554 4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857558 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857562 4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857566 4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857570 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857574 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857578 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857583 4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857587 4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857592 4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857598 4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857604 4924 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857610 4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857615 4924 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857619 4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857624 4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857629 4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857634 4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:26.870863 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857639 4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857650 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857654 4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857658 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857663 4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857667 4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857671 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857675 4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857679 4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857683 4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857687 4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857692 4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857696 4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857702 4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857707 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857712 4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857716 4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857721 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857725 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:26.871528 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857729 4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857733 4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857737 4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857742 4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857747 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857751 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857755 4924 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857759 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857763 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857768 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857772 4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857776 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857780 4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857784 4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857789 4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857799 4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:26.872122 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857804 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.857841 4924 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857898 4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857903 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857908 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857912 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857916 4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857921 4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857925 4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857932 4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857938 4924 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857943 4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857948 4924 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857953 4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857958 4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857962 4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:26.872609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857967 4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857971 4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857975 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857979 4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857984 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857988 4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857992 4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.857996 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858000 4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858004 4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858008 4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858013 4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858017 4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858021 4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858027 4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858032 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858044 4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858048 4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858053 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:26.873056 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858057 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858062 4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858066 4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858084 4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858089 4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858093 4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858097 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858101 4924 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858105 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858109 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858113 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858118 4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858123 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858127 4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858131 4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858136 4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858140 4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858144 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858149 4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858153 4924 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:26.873652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858157 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858161 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858165 4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858170 4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858174 4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858178 4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858182 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858186 4924 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858190 4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858194 4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858209 4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858213 4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858218 4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858222 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858226 4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858230 4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858234 4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858238 4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858243 4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:26.874450 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858247 4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858251 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858255 4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858260 4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858264 4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858268 4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858272 4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858276 4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858280 4924 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858284 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858291 4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858296 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858300 4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858304 4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858308 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858313 4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858317 4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858321 4924 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858325 4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:26.875147 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858331 4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858335 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858340 4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858344 4924 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858348 4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858359 4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858365 4924 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858370 4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858374 4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858378 4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858382 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858386 4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858390 4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858394 4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:26.858398 4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.858509 4924 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.860062 4924 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.860144 4924 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.860406 4924 server.go:1014] "Starting client certificate rotation" Nov 03 13:04:26.875829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.860550 4924 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:04:26.876762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.860612 4924 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 12:58:44 +0000 UTC" deadline="2028-05-02 18:53:49.873123321 +0000 UTC" Nov 03 13:04:26.876762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.860641 4924 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13109h49m23.012484204s" Nov 03 13:04:26.876762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.861172 4924 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:04:26.876762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.862921 4924 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:04:26.876762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.864413 4924 log.go:25] "Validated CRI v1 runtime API" Nov 03 13:04:26.876762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.866701 4924 log.go:25] "Validated CRI v1 image API" Nov 03 13:04:26.876762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.867724 4924 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 13:04:26.876762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.870539 4924 fs.go:135] Filesystem UUIDs: map[1068437d-1ad0-47b3-b577-4f35c644470f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 987425b1-716e-4549-9735-e8109ec8d371:/dev/nvme0n1p4] Nov 03 13:04:26.877083 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.870566 4924 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07e63683dd8aee44f4ecc697513b5cb1b6ad02fcb6eb5172555599a5df33a7ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07e63683dd8aee44f4ecc697513b5cb1b6ad02fcb6eb5172555599a5df33a7ca/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/083d68b9024a7349f0bfe6d845b4d847dfb1d732c4910ebe1bca351cdf28db04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/083d68b9024a7349f0bfe6d845b4d847dfb1d732c4910ebe1bca351cdf28db04/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19202b440d5351427cbaba3d74b7f2e3028bc94d1ad348b0d34a0f4ec9a30da5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19202b440d5351427cbaba3d74b7f2e3028bc94d1ad348b0d34a0f4ec9a30da5/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c90366672e0ef27d9dd4dbf0f23da3c12550a28be185bb6ac5ebb5a3b4970e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c90366672e0ef27d9dd4dbf0f23da3c12550a28be185bb6ac5ebb5a3b4970e2/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d2fe075bfbd3e9985a88fbc3db540d5726c9f1e6881e7793b33bf4d34850e4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d2fe075bfbd3e9985a88fbc3db540d5726c9f1e6881e7793b33bf4d34850e4e/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3148eac97ea1103d424f88bf6b628e4061236c7ab2a3a51cb556742cb08562cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3148eac97ea1103d424f88bf6b628e4061236c7ab2a3a51cb556742cb08562cf/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33c50d8bc8a219dcd53899ebd8d38b978b783083de9739e94bde74167b801ff2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33c50d8bc8a219dcd53899ebd8d38b978b783083de9739e94bde74167b801ff2/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35f342274cdc3da99c9116e872a17dae775a0217c13fe74d46312f367b1bf550/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35f342274cdc3da99c9116e872a17dae775a0217c13fe74d46312f367b1bf550/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/377e7a9511a7951a0a720ce9c9f433697c003679361aecdcca5ba678f586af84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/377e7a9511a7951a0a720ce9c9f433697c003679361aecdcca5ba678f586af84/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d8e2658eb5bb2e05bd690d7010519ef355254e88c0cd998348c646617a744bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d8e2658eb5bb2e05bd690d7010519ef355254e88c0cd998348c646617a744bb/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cb6b2f6020a0a6bd4c97e359bae474d91ec5efcfce73a0bd9e5a7a250595058/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cb6b2f6020a0a6bd4c97e359bae474d91ec5efcfce73a0bd9e5a7a250595058/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fabe995f2d02e05bd2f600b5ad1d727e90c8ccff06d35d24bba5eafe2f5ee48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fabe995f2d02e05bd2f600b5ad1d727e90c8ccff06d35d24bba5eafe2f5ee48/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6446669ac9b0972e5f31a23ca710736a72b640a9d9e0d93475d3beef4a08496f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6446669ac9b0972e5f31a23ca710736a72b640a9d9e0d93475d3beef4a08496f/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8448ca709f038b2765a1970363c93fe64b5901421a1d46858280ed444f73d7cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8448ca709f038b2765a1970363c93fe64b5901421a1d46858280ed444f73d7cb/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/873553354e91eee835b2c8273a8c51091b8df8cb25707826cfbfba6d9c35ec12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/873553354e91eee835b2c8273a8c51091b8df8cb25707826cfbfba6d9c35ec12/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d49330c440a72efe7f3c3950b2c0ccd385e74f9b274e4e4011b884969259e9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d49330c440a72efe7f3c3950b2c0ccd385e74f9b274e4e4011b884969259e9b/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/909314a560bfc46fb1d79e8ec5dc77c4f44c3546b0243b435193b8bd0a285f22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/909314a560bfc46fb1d79e8ec5dc77c4f44c3546b0243b435193b8bd0a285f22/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96ea355720fe72a5adc432280a5160a0bd7fc9c5c79c5f89c8171c358556ed91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96ea355720fe72a5adc432280a5160a0bd7fc9c5c79c5f89c8171c358556ed91/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99f4aca45c64a74d0f9897e4f4f028ef82c681cd3d5b7c1bf07a0ceca20ba127/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99f4aca45c64a74d0f9897e4f4f028ef82c681cd3d5b7c1bf07a0ceca20ba127/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7ba1202a628a1ad92797c9abcc8229acf4de3e7d09078afee555fbbb1288bb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7ba1202a628a1ad92797c9abcc8229acf4de3e7d09078afee555fbbb1288bb5/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc857669bc04ba5000b5a232dc118f91b3248463f19cff875d13c7711cc6844d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc857669bc04ba5000b5a232dc118f91b3248463f19cff875d13c7711cc6844d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8b8a280f1ea9d9f11e039953b6bf403ecdb5f34fc62d4493024756e8f9c208d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8b8a280f1ea9d9f11e039953b6bf403ecdb5f34fc62d4493024756e8f9c208d/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9fc03dda4bf0721a969454a433c5bf7f47fbeaf5212fc5344e423f6846b4fbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9fc03dda4bf0721a969454a433c5bf7f47fbeaf5212fc5344e423f6846b4fbf/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05516dd7-e72f-4b64-9707-e401c7e6198b/volumes/kubernetes.io~projected/kube-api-access-cpfgn:{mountpoint:/var/lib/kubelet/pods/05516dd7-e72f-4b64-9707-e401c7e6198b/volumes/kubernetes.io~projected/kube-api-access-cpfgn major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08ac07c7-d646-4d10-8ba3-844c752e3a37/volumes/kubernetes.io~projected/kube-api-access-xqn2s:{mountpoint:/var/lib/kubelet/pods/08ac07c7-d646-4d10-8ba3-844c752e3a37/volumes/kubernetes.io~projected/kube-api-access-xqn2s major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1651b048-50ba-4391-8382-60f9cffd1fa7/volumes/kubernetes.io~projected/kube-api-access-mhbkm:{mountpoint:/var/lib/kubelet/pods/1651b048-50ba-4391-8382-60f9cffd1fa7/volumes/kubernetes.io~projected/kube-api-access-mhbkm major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f645ece-f5a0-478c-a0a8-a772c740a072/volumes/kubernetes.io~projected/kube-api-access-s6ds8:{mountpoint:/var/lib/kubelet/pods/1f645ece-f5a0-478c-a0a8-a772c740a072/volumes/kubernetes.io~projected/kube-api-access-s6ds8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/249644e9-11b8-48ad-8e70-79e3b87c550a/volumes/kubernetes.io~projected/kube-api-access-btxph:{mountpoint:/var/lib/kubelet/pods/249644e9-11b8-48ad-8e70-79e3b87c550a/volumes/kubernetes.io~projected/kube-api-access-btxph major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2/volumes/kubernetes.io~projected/kube-api-access-gszmj:{mountpoint:/var/lib/kubelet/pods/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2/volumes/kubernetes.io~projected/kube-api-access-gszmj major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970/volumes/kubernetes.io~projected/kube-api-access-bp7fc:{mountpoint:/var/lib/kubelet/pods/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970/volumes/kubernetes.io~projected/kube-api-access-bp7fc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50d1326a-e89d-4e45-90c6-9a254a157e48/volumes/kubernetes.io~projected/kube-api-access-79g67:{mountpoint:/var/lib/kubelet/pods/50d1326a-e89d-4e45-90c6-9a254a157e48/volumes/kubernetes.io~projected/kube-api-access-79g67 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50d1326a-e89d-4e45-90c6-9a254a157e48/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/50d1326a-e89d-4e45-90c6-9a254a157e48/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d/volumes/kubernetes.io~projected/kube-api-access-q822s:{mountpoint:/var/lib/kubelet/pods/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d/volumes/kubernetes.io~projected/kube-api-access-q822s major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53c4e88e-1203-4083-98e9-9a2202a3a704/volumes/kubernetes.io~projected/kube-api-access-kjwcb:{mountpoint:/var/lib/kubelet/pods/53c4e88e-1203-4083-98e9-9a2202a3a704/volumes/kubernetes.io~projected/kube-api-access-kjwcb major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/541aac48-8aa9-40e2-88ec-17687fdd5167/volumes/kubernetes.io~projected/kube-api-access-rfz2t:{mountpoint:/var/lib/kubelet/pods/541aac48-8aa9-40e2-88ec-17687fdd5167/volumes/kubernetes.io~projected/kube-api-access-rfz2t major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/541aac48-8aa9-40e2-88ec-17687fdd5167/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/541aac48-8aa9-40e2-88ec-17687fdd5167/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~projected/kube-api-access-znzwn:{mountpoint:/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~projected/kube-api-access-znzwn major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6365006c-9e6f-406f-9102-e3b0b452675d/volumes/kubernetes.io~projected/kube-api-access-2289v:{mountpoint:/var/lib/kubelet/pods/6365006c-9e6f-406f-9102-e3b0b452675d/volumes/kubernetes.io~projected/kube-api-access-2289v major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c984d62-74a3-4dab-9639-8fed18759ddc/volumes/kubernetes.io~projected/kube-api-access-lgdd6:{mountpoint:/var/lib/kubelet/pods/6c984d62-74a3-4dab-9639-8fed18759ddc/volumes/kubernetes.io~projected/kube-api-access-lgdd6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71f66428-6b35-45cc-ac46-ce5c07aa958c/volumes/kubernetes.io~projected/kube-api-access-67qc5:{mountpoint:/var/lib/kubelet/pods/71f66428-6b35-45cc-ac46-ce5c07aa958c/volumes/kubernetes.io~projected/kube-api-access-67qc5 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71f66428-6b35-45cc-ac46-ce5c07aa958c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/71f66428-6b35-45cc-ac46-ce5c07aa958c/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70/volumes/kubernetes.io~projected/kube-api-access-trds9:{mountpoint:/var/lib/kubelet/pods/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70/volumes/kubernetes.io~projected/kube-api-access-trds9 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d1d0d87-bbca-4424-bfad-d0b312732700/volumes/kubernetes.io~projected/kube-api-access-s2zf7:{mountpoint:/var/lib/kubelet/pods/7d1d0d87-bbca-4424-bfad-d0b312732700/volumes/kubernetes.io~projected/kube-api-access-s2zf7 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8425f3a6-d496-41eb-bb83-c65f6011890b/volumes/kubernetes.io~projected/kube-api-access-jw7h6:{mountpoint:/var/lib/kubelet/pods/8425f3a6-d496-41eb-bb83-c65f6011890b/volumes/kubernetes.io~projected/kube-api-access-jw7h6 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99ae820d-6217-425b-9451-247f5dff70dc/volumes/kubernetes.io~projected/kube-api-access-l9htq:{mountpoint:/var/lib/kubelet/pods/99ae820d-6217-425b-9451-247f5dff70dc/volumes/kubernetes.io~projected/kube-api-access-l9htq major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3/volumes/kubernetes.io~projected/kube-api-access-rgtt9:{mountpoint:/var/lib/kubelet/pods/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3/volumes/kubernetes.io~projected/kube-api-access-rgtt9 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~projected/kube-api-access-btfds:{mountpoint:/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~projected/kube-api-access-btfds major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc64bc02-50f7-4594-8eaa-3a4ec97763f1/volumes/kubernetes.io~projected/kube-api-access-6nc4l:{mountpoint:/var/lib/kubelet/pods/cc64bc02-50f7-4594-8eaa-3a4ec97763f1/volumes/kubernetes.io~projected/kube-api-access-6nc4l major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a/volumes/kubernetes.io~projected/kube-api-access-74697:{mountpoint:/var/lib/kubelet/pods/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a/volumes/kubernetes.io~projected/kube-api-access-74697 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~projected/kube-api-access-cvbch:{mountpoint:/var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~projected/kube-api-access-cvbch major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volumes/kubernetes.io~projected/kube-api-access-thp2p:{mountpoint:/var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volumes/kubernetes.io~projected/kube-api-access-thp2p major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec0b3420-306f-4a10-95ff-0bcf4a4f1730/volumes/kubernetes.io~projected/kube-api-access-hccn5:{mountpoint:/var/lib/kubelet/pods/ec0b3420-306f-4a10-95ff-0bcf4a4f1730/volumes/kubernetes.io~projected/kube-api-access-hccn5 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eed1b9f7-3769-46a0-b8ba-63eb8d17a952/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/eed1b9f7-3769-46a0-b8ba-63eb8d17a952/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~projected/kube-api-access-4rnmt:{mountpoint:/var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~projected/kube-api-access-4rnmt major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f70a3801-ffa0-4049-aeee-0d886ccb30b3/volumes/kubernetes.io~projected/kube-api-access-dl2z6:{mountpoint:/var/lib/kubelet/pods/f70a3801-ffa0-4049-aeee-0d886ccb30b3/volumes/kubernetes.io~projected/kube-api-access-dl2z6 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f70a3801-ffa0-4049-aeee-0d886ccb30b3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f70a3801-ffa0-4049-aeee-0d886ccb30b3/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/554e8266fc7e406fa89abb1548c628be60677012114d0f2d85ca5cdab2bd0288/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/931dcf1463abfffb3c64ad0bc91ef3ae19c7d9e8460d483d2aa434b37ea7ffc0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/16a0b8044f8a22a16f1faf0cbaa6df1c2ca0118b7604dad2736d7908966a9fc5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/14ee1e670dabee50ac422dee66eb2e75e9fa3db0dda58f257cc20c9029a7335c/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/eee356943f785e14436284f00afd6400712dbbc43725e26a965f835ef4e35e71/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b8f304f1f8a8e495e7b21570b1344dc02b868a3d558e38a72aeb623c1f8e2374/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9ef720fefd8bfb5860650fb2b666ed5063edc067683ec10a3cb849bf0b56ab09/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/eb521a0a8cf16aeea1776eb945d504d456d7c2884c728eb67c308fd93ee1a3a1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/f4e0d7c460813d550bafc84b63e127397b931ba567b7db021d5fd310d87ecdcc/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ec2de6cd40733c18f18eb8f1abd57ea378f7f2940434cb59062ec1f15b9092be/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/90d21aae2fa0f6bb08c026f8bfb41a6da64bcf20530558154a68b0326efbeea0/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5ad2558b75d844f83c45c56661af1a01b976e0285eaa5a8d24f26083fa3caa13/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/55f006ee0c25b0eec686a0f838c6267503a5e640a40b6b1fb083ed8da5331730/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b7527d2a0a31f64d3496845aad0bb787dae8e85fddf231520e07108de1d344ca/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/32940c451056fad8594a2721cc23218db65e34f39e54d646acf2d486bcfda5d3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e91e740913d0def27c903b01e2fd6344808f3b11b013166bab35a674b6ec5bf6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/395aaa208aa2ad10a8d1b87da52ccdbda1166d7adf9a0784a39016be122b6967/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c1bce98aa201e8a84f5135e07cca01e2c456c44e1a85c2167a5219b4af88b666/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/68e8b9e4536dcc266a8da9c10f82e404022eed6636484125d1aa23510b1f3997/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b839f7c19b24ab1127c38eb79d0bbe0ce47108c180d0c9c30b945b007eb6fcce/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/15dd7b2a33775a161c78cdd988a8ad885973baed5b06ccb31c9a0d55cf9ccc0e/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d5906f59836e01019c7fd9680274a951fcb19b39cd8c15f4ea9d71a073abb737/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/81bc4ed07620775d98362e4b38cc94a6adf1880d39439adba89891cc649b49fe/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/6ed10539132e145716ddf375b4945349e6b1888552d96747da4459d3c72df11c/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a6497849371e578c5003f0c4833bd47969b1646555a544a951030dc9253d9209/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/b9af1316066a954bbe1fb7dd503f76f11a8df993521078703bc0aab79fe4c068/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/2aae7141075de236cc47c5fa30519a25131f9aa01dd38707f6ac894e33ff911b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/21f76b9d6c036e402ec4f227887ac4856db2f3c0f760d0d03f1f5fe6b806cd1d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/bb9311eeec492e8d9622451e3ff3c29022e6d6ab08ff2dadc61cd56e081bf55f/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/10748eb6118a90fb618781b7906b490bfbd6d92ad3b55439bb0b0e3836a3b611/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/e5d1f7cd087bce6364a4b3f0d711b1df4fa828dc04a465fe3d528e3b529dad9f/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/ee0260818d5b320702e39ca7e380ccabfc01f3b84846d169baf1bba76468a936/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/8986e7ddab39ae0ced0437fdf890b5ffa834f4025cda037ce1f0d1128ea1c606/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/27d9576c063948d4fbbc325fe642581cdcd4ce35e4351ec0a117ce35647d93b7/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/6974c07dd902f0e398a43bdae4d387a5b6994869615c54999bb1b7c7910b64db/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/40d596572eaaf3e8cf7267c3d66d16c9b58e6e111446d33e3638021031513207/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/006329e9462ad90a51947db09051928e67aecd653ec8e894a5738c6e6a17ad41/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/65d6a1b68f120ab833c27a8c4283dc44d84ba8a52208b57dd4c8a600196fd524/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d588d436e3f520cc0b838310ff1b8465fd902f0659a06c6c4070ae0c512542d0/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/60246f8216a854c37812b286e789aa7f3f0b034213d9670d133a8a766011df36/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/86cea5ae58623a7792747f6b4b24baa7830f7d9bd652ec36c730b80eed276064/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/8919e1acbc3df22111d3982fbc62df01dcfed9a13a6d006367ade685d87d6daf/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/6411c107e9805cde8c2e7eff5b5ee9e6e5f39ac03ac8fb872ac540d5f7f22e61/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/90120abf94e54abc81b87c3c9116b02f6d4a15532bcf39c88467c45d1abd81e9/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/58886200044f420e50cac3aeb18e59c05d4fefe3b9486964f72004fdde174df0/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b830751bdc0c19d65026c2590563d5bb43d988b463dbc8ceb7cbe9db0bee594b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/894532136ec52800e18273a797939ac07d693bcfa8df20409ff9528f0f81cb6a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/671871fd3553346061e97139c4096748e15eaf31232f88dabc350ecd6485911d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3be62b5a3a9d7ce6c2ea119de8addcd5ba8814b991defad5a63d528c25ac15bc/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/24f7ceac159da6a27e602a0da5d7214e0ddf975424eddadabf75a5c25bc9f294/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/1ae2a9bcaabf023af4a5ef573f6b1c7b3e12c4fb0f85cbb4902596e38c192cb8/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/6541b8b9745f9c63ae64e003bbc9224979f254ce01ab7152b36f8ce67085c2eb/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/11dfe0ebbebb0f33cfe2444e80374d113a39cc0ee2ff3db9bd497f3d1008392b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d5fe8832f5d9598f3471eb99b71874e8c13a55d4345b5084c3590be456182977/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/57600435ac03e24e579b0e32eb764b0586cab6f453c7f04d2cbbb2d1139ae1c9/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 13:04:26.882482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.881845 4924 manager.go:217] Machine: {Timestamp:2026-11-03 13:04:26.880980107 +0000 UTC m=+0.075383696 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099172 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22aba239f9628c35cec9706a235fb5 SystemUUID:ec22aba2-39f9-628c-35ce-c9706a235fb5 BootID:3a59fc5f-7f52-487c-b830-5dcf1f9b0cb5 Filesystems:[{Device:/var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volumes/kubernetes.io~projected/kube-api-access-thp2p DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ec0b3420-306f-4a10-95ff-0bcf4a4f1730/volumes/kubernetes.io~projected/kube-api-access-hccn5 DeviceMajor:0 DeviceMinor:224 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71f66428-6b35-45cc-ac46-ce5c07aa958c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70/volumes/kubernetes.io~projected/kube-api-access-trds9 DeviceMajor:0 DeviceMinor:226 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d90132-40a2-4f02-8f59-e0059a1c0a1a/volumes/kubernetes.io~projected/kube-api-access-cvbch DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99f4aca45c64a74d0f9897e4f4f028ef82c681cd3d5b7c1bf07a0ceca20ba127/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8448ca709f038b2765a1970363c93fe64b5901421a1d46858280ed444f73d7cb/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33c50d8bc8a219dcd53899ebd8d38b978b783083de9739e94bde74167b801ff2/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50d1326a-e89d-4e45-90c6-9a254a157e48/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2/volumes/kubernetes.io~projected/kube-api-access-gszmj DeviceMajor:0 DeviceMinor:220 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~projected/kube-api-access-btfds DeviceMajor:0 DeviceMinor:223 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d2fe075bfbd3e9985a88fbc3db540d5726c9f1e6881e7793b33bf4d34850e4e/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96ea355720fe72a5adc432280a5160a0bd7fc9c5c79c5f89c8171c358556ed91/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6365006c-9e6f-406f-9102-e3b0b452675d/volumes/kubernetes.io~projected/kube-api-access-2289v DeviceMajor:0 DeviceMinor:212 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/05516dd7-e72f-4b64-9707-e401c7e6198b/volumes/kubernetes.io~projected/kube-api-access-cpfgn DeviceMajor:0 DeviceMinor:222 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7ba1202a628a1ad92797c9abcc8229acf4de3e7d09078afee555fbbb1288bb5/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1651b048-50ba-4391-8382-60f9cffd1fa7/volumes/kubernetes.io~projected/kube-api-access-mhbkm DeviceMajor:0 DeviceMinor:279 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~projected/kube-api-access-znzwn DeviceMajor:0 DeviceMinor:216 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/08ac07c7-d646-4d10-8ba3-844c752e3a37/volumes/kubernetes.io~projected/kube-api-access-xqn2s DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9fc03dda4bf0721a969454a433c5bf7f47fbeaf5212fc5344e423f6846b4fbf/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d8e2658eb5bb2e05bd690d7010519ef355254e88c0cd998348c646617a744bb/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8b8a280f1ea9d9f11e039953b6bf403ecdb5f34fc62d4493024756e8f9c208d/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35f342274cdc3da99c9116e872a17dae775a0217c13fe74d46312f367b1bf550/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f70a3801-ffa0-4049-aeee-0d886ccb30b3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc64bc02-50f7-4594-8eaa-3a4ec97763f1/volumes/kubernetes.io~projected/kube-api-access-6nc4l DeviceMajor:0 DeviceMinor:335 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/249644e9-11b8-48ad-8e70-79e3b87c550a/volumes/kubernetes.io~projected/kube-api-access-btxph DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3/volumes/kubernetes.io~projected/kube-api-access-rgtt9 DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d1d0d87-bbca-4424-bfad-d0b312732700/volumes/kubernetes.io~projected/kube-api-access-s2zf7 DeviceMajor:0 DeviceMinor:217 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07e63683dd8aee44f4ecc697513b5cb1b6ad02fcb6eb5172555599a5df33a7ca/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc857669bc04ba5000b5a232dc118f91b3248463f19cff875d13c7711cc6844d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6c984d62-74a3-4dab-9639-8fed18759ddc/volumes/kubernetes.io~projected/kube-api-access-lgdd6 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dd77bb7b-1f79-4021-a9ad-70160e72cda4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d/volumes/kubernetes.io~projected/kube-api-access-q822s DeviceMajor:0 DeviceMinor:225 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a/volumes/kubernetes.io~projected/kube-api-access-74697 DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/541aac48-8aa9-40e2-88ec-17687fdd5167/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/99ae820d-6217-425b-9451-247f5dff70dc/volumes/kubernetes.io~projected/kube-api-access-l9htq DeviceMajor:0 DeviceMinor:237 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/377e7a9511a7951a0a720ce9c9f433697c003679361aecdcca5ba678f586af84/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/873553354e91eee835b2c8273a8c51091b8df8cb25707826cfbfba6d9c35ec12/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/53c4e88e-1203-4083-98e9-9a2202a3a704/volumes/kubernetes.io~projected/kube-api-access-kjwcb DeviceMajor:0 DeviceMinor:218 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d49330c440a72efe7f3c3950b2c0ccd385e74f9b274e4e4011b884969259e9b/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/541aac48-8aa9-40e2-88ec-17687fdd5167/volumes/kubernetes.io~projected/kube-api-access-rfz2t DeviceMajor:0 DeviceMinor:219 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eed1b9f7-3769-46a0-b8ba-63eb8d17a952/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:278 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6446669ac9b0972e5f31a23ca710736a72b640a9d9e0d93475d3beef4a08496f/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/909314a560bfc46fb1d79e8ec5dc77c4f44c3546b0243b435193b8bd0a285f22/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/083d68b9024a7349f0bfe6d845b4d847dfb1d732c4910ebe1bca351cdf28db04/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c90366672e0ef27d9dd4dbf0f23da3c12550a28be185bb6ac5ebb5a3b4970e2/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19202b440d5351427cbaba3d74b7f2e3028bc94d1ad348b0d34a0f4ec9a30da5/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/volumes/kubernetes.io~projected/kube-api-access-4rnmt DeviceMajor:0 DeviceMinor:215 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71f66428-6b35-45cc-ac46-ce5c07aa958c/volumes/kubernetes.io~projected/kube-api-access-67qc5 DeviceMajor:0 DeviceMinor:221 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970/volumes/kubernetes.io~projected/kube-api-access-bp7fc DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3148eac97ea1103d424f88bf6b628e4061236c7ab2a3a51cb556742cb08562cf/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1f645ece-f5a0-478c-a0a8-a772c740a072/volumes/kubernetes.io~projected/kube-api-access-s6ds8 DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f70a3801-ffa0-4049-aeee-0d886ccb30b3/volumes/kubernetes.io~projected/kube-api-access-dl2z6 DeviceMajor:0 DeviceMinor:211 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50d1326a-e89d-4e45-90c6-9a254a157e48/volumes/kubernetes.io~projected/kube-api-access-79g67 DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cb6b2f6020a0a6bd4c97e359bae474d91ec5efcfce73a0bd9e5a7a250595058/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fabe995f2d02e05bd2f600b5ad1d727e90c8ccff06d35d24bba5eafe2f5ee48/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8425f3a6-d496-41eb-bb83-c65f6011890b/volumes/kubernetes.io~projected/kube-api-access-jw7h6 DeviceMajor:0 DeviceMinor:318 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:19202b440d53514 MacAddress:1a:7b:e3:11:96:09 Speed:10000 Mtu:8901} {Name:2d2fe075bfbd3e9 MacAddress:02:c2:fe:62:1d:46 Speed:10000 Mtu:8901} {Name:3148eac97ea1103 MacAddress:0e:4a:d3:6c:6d:61 Speed:10000 Mtu:8901} {Name:33c50d8bc8a219d MacAddress:86:f5:16:e0:c3:55 Speed:10000 Mtu:8901} {Name:35f342274cdc3da MacAddress:1e:37:02:e4:b6:d9 Speed:10000 Mtu:8901} {Name:377e7a9511a7951 MacAddress:de:96:98:be:a7:e8 Speed:10000 Mtu:8901} {Name:3d8e2658eb5bb2e MacAddress:c2:79:ce:1f:53:36 Speed:10000 Mtu:8901} {Name:6446669ac9b0972 MacAddress:b2:1b:47:a3:38:e5 Speed:10000 Mtu:8901} {Name:8448ca709f038b2 MacAddress:2a:3a:53:84:24:b4 Speed:10000 Mtu:8901} {Name:8d49330c440a72e MacAddress:2a:fe:b7:9c:37:c9 Speed:10000 Mtu:8901} {Name:909314a560bfc46 MacAddress:42:41:97:28:1a:59 Speed:10000 Mtu:8901} {Name:99f4aca45c64a74 MacAddress:d6:09:b2:2b:29:23 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c3:ea:31:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:39:73:93:94:e1 Speed:0 Mtu:8901} {Name:d8b8a280f1ea9d9 MacAddress:0e:74:f7:27:88:4e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c3:ea:31:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:be:a0:b5:af:45:3f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:05:94:8f:55:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 13:04:26.882733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.882497 4924 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 13:04:26.882733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.882562 4924 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 13:04:26.882923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.882900 4924 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 13:04:26.883132 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.882926 4924 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-63.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 13:04:26.883196 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883135 4924 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 13:04:26.883196 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883148 4924 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 13:04:26.883196 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883163 4924 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:04:26.883196 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883185 4924 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:04:26.883309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883211 4924 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 13:04:26.883424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883413 4924 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 13:04:26.883575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883564 4924 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 13:04:26.883618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883609 4924 kubelet.go:493] "Attempting to sync node with API server" Nov 03 13:04:26.883651 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883625 4924 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 13:04:26.883651 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883639 4924 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 13:04:26.883719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883654 4924 kubelet.go:405] "Adding apiserver pod source" Nov 03 13:04:26.883719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.883666 4924 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 13:04:26.884608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.884590 4924 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 13:04:26.884608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.884604 4924 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:04:26.885143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.885123 4924 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 13:04:26.897593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.897573 4924 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 13:04:26.897593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.897593 4924 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:04:26.897729 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.897715 4924 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 13:04:26.897954 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.897941 4924 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 13:04:26.897994 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.897972 4924 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 13:04:26.898096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898085 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 13:04:26.898134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898103 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 13:04:26.898134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898109 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 13:04:26.898134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898116 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 13:04:26.898134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898122 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 13:04:26.898134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898127 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 13:04:26.898134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898133 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 13:04:26.898369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898138 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 13:04:26.898369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898152 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 13:04:26.898369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898162 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 13:04:26.898369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898177 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 13:04:26.898369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898195 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 13:04:26.898369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898228 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 13:04:26.898369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.898235 4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 13:04:26.902081 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.902054 4924 server.go:1263] "Started kubelet" Nov 03 13:04:26.904206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.902145 4924 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 13:04:26.904206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.902586 4924 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 13:04:26.904206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.902677 4924 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 13:04:26.904206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.902981 4924 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 13:04:26.904206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.904013 4924 server.go:317] "Adding debug handlers to kubelet server" Nov 03 13:04:26.902724 ip-10-0-129-63 systemd[1]: Started Kubernetes Kubelet. Nov 03 13:04:26.908576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.908205 4924 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 13:04:26.908576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.908415 4924 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 13:04:26.908771 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.908589 4924 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:58:46 +0000 UTC" deadline="2028-08-18 00:03:40.905128196 +0000 UTC" Nov 03 13:04:26.908771 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.908645 4924 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15682h59m13.996487784s" Nov 03 13:04:26.908939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.908784 4924 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 13:04:26.908939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.908795 4924 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 13:04:26.909786 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:26.909761 4924 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:04:26.909786 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.909781 4924 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 13:04:26.911312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.911292 4924 factory.go:153] Registering CRI-O factory Nov 03 13:04:26.911312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.911314 4924 factory.go:223] Registration of the crio container factory successfully Nov 03 13:04:26.911487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.911368 4924 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 13:04:26.911487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.911378 4924 factory.go:55] Registering systemd factory Nov 03 13:04:26.911487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.911385 4924 factory.go:223] Registration of the systemd container factory successfully Nov 03 13:04:26.911487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.911402 4924 factory.go:103] Registering Raw factory Nov 03 13:04:26.911487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.911417 4924 manager.go:1199] Started watching for new ooms in manager Nov 03 13:04:26.912017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.911775 4924 manager.go:319] Starting recovery of all containers Nov 03 13:04:26.913444 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913396 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ac07c7-d646-4d10-8ba3-844c752e3a37" volumeName="kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-daemon-config" seLinuxMountContext="" Nov 03 13:04:26.913444 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913434 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c984d62-74a3-4dab-9639-8fed18759ddc" volumeName="kubernetes.io/projected/6c984d62-74a3-4dab-9639-8fed18759ddc-kube-api-access-lgdd6" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913449 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d1d0d87-bbca-4424-bfad-d0b312732700" volumeName="kubernetes.io/projected/7d1d0d87-bbca-4424-bfad-d0b312732700-kube-api-access-s2zf7" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913462 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8cd572f-aee9-4f80-87c2-2504a0ba404d" volumeName="kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913475 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8cd572f-aee9-4f80-87c2-2504a0ba404d" volumeName="kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913500 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec0b3420-306f-4a10-95ff-0bcf4a4f1730" volumeName="kubernetes.io/projected/ec0b3420-306f-4a10-95ff-0bcf4a4f1730-kube-api-access-hccn5" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913514 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11da27b-1a3d-4fca-b78e-fe9cd3caa46b" volumeName="kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-default-certificate" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913526 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f70a3801-ffa0-4049-aeee-0d886ccb30b3" volumeName="kubernetes.io/projected/f70a3801-ffa0-4049-aeee-0d886ccb30b3-kube-api-access-dl2z6" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913540 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b7eaba1-903d-4c79-9cb9-f0ba75d8b970" volumeName="kubernetes.io/projected/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-kube-api-access-bp7fc" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913551 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53c4e88e-1203-4083-98e9-9a2202a3a704" volumeName="kubernetes.io/projected/53c4e88e-1203-4083-98e9-9a2202a3a704-kube-api-access-kjwcb" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913565 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="541aac48-8aa9-40e2-88ec-17687fdd5167" volumeName="kubernetes.io/secret/541aac48-8aa9-40e2-88ec-17687fdd5167-cert" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913578 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56168f93-9536-4108-a5ce-5677c95ecf0c" volumeName="kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913600 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f66428-6b35-45cc-ac46-ce5c07aa958c" volumeName="kubernetes.io/configmap/71f66428-6b35-45cc-ac46-ce5c07aa958c-config" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913615 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d90132-40a2-4f02-8f59-e0059a1c0a1a" volumeName="kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-tuned" seLinuxMountContext="" Nov 03 13:04:26.913630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913630 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f70a3801-ffa0-4049-aeee-0d886ccb30b3" volumeName="kubernetes.io/secret/f70a3801-ffa0-4049-aeee-0d886ccb30b3-serving-cert" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913642 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="249644e9-11b8-48ad-8e70-79e3b87c550a" volumeName="kubernetes.io/empty-dir/249644e9-11b8-48ad-8e70-79e3b87c550a-tmp-dir" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913655 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2" volumeName="kubernetes.io/projected/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-kube-api-access-gszmj" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913668 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b2bd33-97be-42ee-b7d4-2b86f1de3d7d" volumeName="kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-service-ca-bundle" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913691 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc" volumeName="kubernetes.io/configmap/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-konnectivity-ca" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913704 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd77bb7b-1f79-4021-a9ad-70160e72cda4" volumeName="kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-script-lib" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913718 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2" volumeName="kubernetes.io/configmap/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-telemetry-config" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913731 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b2bd33-97be-42ee-b7d4-2b86f1de3d7d" volumeName="kubernetes.io/projected/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-kube-api-access-q822s" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913743 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c984d62-74a3-4dab-9639-8fed18759ddc" volumeName="kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-binary-copy" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913757 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8425f3a6-d496-41eb-bb83-c65f6011890b" volumeName="kubernetes.io/empty-dir/8425f3a6-d496-41eb-bb83-c65f6011890b-data-volume" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913770 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d90132-40a2-4f02-8f59-e0059a1c0a1a" volumeName="kubernetes.io/projected/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-kube-api-access-cvbch" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913782 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d1326a-e89d-4e45-90c6-9a254a157e48" volumeName="kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-trusted-ca" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913794 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d1326a-e89d-4e45-90c6-9a254a157e48" volumeName="kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-config" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913805 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd77bb7b-1f79-4021-a9ad-70160e72cda4" volumeName="kubernetes.io/projected/dd77bb7b-1f79-4021-a9ad-70160e72cda4-kube-api-access-thp2p" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913836 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1651b048-50ba-4391-8382-60f9cffd1fa7" volumeName="kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913859 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6365006c-9e6f-406f-9102-e3b0b452675d" volumeName="kubernetes.io/projected/6365006c-9e6f-406f-9102-e3b0b452675d-kube-api-access-2289v" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913871 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78248e2d-8c0b-47fb-b55b-07e0ffdc1c70" volumeName="kubernetes.io/projected/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70-kube-api-access-trds9" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913882 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8425f3a6-d496-41eb-bb83-c65f6011890b" volumeName="kubernetes.io/projected/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-api-access-jw7h6" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913895 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ae820d-6217-425b-9451-247f5dff70dc" volumeName="kubernetes.io/empty-dir/99ae820d-6217-425b-9451-247f5dff70dc-tmp-dir" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913906 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b2bd33-97be-42ee-b7d4-2b86f1de3d7d" volumeName="kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-trusted-ca-bundle" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913919 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b2bd33-97be-42ee-b7d4-2b86f1de3d7d" volumeName="kubernetes.io/secret/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-serving-cert" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913933 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8cd572f-aee9-4f80-87c2-2504a0ba404d" volumeName="kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913947 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8cd572f-aee9-4f80-87c2-2504a0ba404d" volumeName="kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913958 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05516dd7-e72f-4b64-9707-e401c7e6198b" volumeName="kubernetes.io/configmap/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-config" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913969 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="249644e9-11b8-48ad-8e70-79e3b87c550a" volumeName="kubernetes.io/projected/249644e9-11b8-48ad-8e70-79e3b87c550a-kube-api-access-btxph" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913980 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d1326a-e89d-4e45-90c6-9a254a157e48" volumeName="kubernetes.io/secret/50d1326a-e89d-4e45-90c6-9a254a157e48-serving-cert" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.913991 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b2bd33-97be-42ee-b7d4-2b86f1de3d7d" volumeName="kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-tmp" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914006 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b2bd33-97be-42ee-b7d4-2b86f1de3d7d" volumeName="kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-snapshots" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914017 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56168f93-9536-4108-a5ce-5677c95ecf0c" volumeName="kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914040 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d90132-40a2-4f02-8f59-e0059a1c0a1a" volumeName="kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-tmp" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914054 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd77bb7b-1f79-4021-a9ad-70160e72cda4" volumeName="kubernetes.io/secret/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914065 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11da27b-1a3d-4fca-b78e-fe9cd3caa46b" volumeName="kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-stats-auth" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914092 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4f67336-02d0-46a8-9389-e4a1e2a80ca6" volumeName="kubernetes.io/configmap/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-nginx-conf" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914104 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ac07c7-d646-4d10-8ba3-844c752e3a37" volumeName="kubernetes.io/projected/08ac07c7-d646-4d10-8ba3-844c752e3a37-kube-api-access-xqn2s" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914116 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b7eaba1-903d-4c79-9cb9-f0ba75d8b970" volumeName="kubernetes.io/configmap/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-serviceca" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914129 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="541aac48-8aa9-40e2-88ec-17687fdd5167" volumeName="kubernetes.io/projected/541aac48-8aa9-40e2-88ec-17687fdd5167-kube-api-access-rfz2t" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914140 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56168f93-9536-4108-a5ce-5677c95ecf0c" volumeName="kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914152 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56168f93-9536-4108-a5ce-5677c95ecf0c" volumeName="kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914165 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c984d62-74a3-4dab-9639-8fed18759ddc" volumeName="kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914183 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8cd572f-aee9-4f80-87c2-2504a0ba404d" volumeName="kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914196 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05516dd7-e72f-4b64-9707-e401c7e6198b" volumeName="kubernetes.io/projected/05516dd7-e72f-4b64-9707-e401c7e6198b-kube-api-access-cpfgn" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914235 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ac07c7-d646-4d10-8ba3-844c752e3a37" volumeName="kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-cni-binary-copy" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914286 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56168f93-9536-4108-a5ce-5677c95ecf0c" volumeName="kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914314 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8cd572f-aee9-4f80-87c2-2504a0ba404d" volumeName="kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914326 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd77bb7b-1f79-4021-a9ad-70160e72cda4" volumeName="kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-env-overrides" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914340 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f66428-6b35-45cc-ac46-ce5c07aa958c" volumeName="kubernetes.io/projected/71f66428-6b35-45cc-ac46-ce5c07aa958c-kube-api-access-67qc5" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914354 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f66428-6b35-45cc-ac46-ce5c07aa958c" volumeName="kubernetes.io/secret/71f66428-6b35-45cc-ac46-ce5c07aa958c-serving-cert" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914367 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ae820d-6217-425b-9451-247f5dff70dc" volumeName="kubernetes.io/configmap/99ae820d-6217-425b-9451-247f5dff70dc-config-volume" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914380 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8cd572f-aee9-4f80-87c2-2504a0ba404d" volumeName="kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914392 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd77bb7b-1f79-4021-a9ad-70160e72cda4" volumeName="kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-config" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914404 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d1326a-e89d-4e45-90c6-9a254a157e48" volumeName="kubernetes.io/projected/50d1326a-e89d-4e45-90c6-9a254a157e48-kube-api-access-79g67" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914418 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56168f93-9536-4108-a5ce-5677c95ecf0c" volumeName="kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914438 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3" volumeName="kubernetes.io/projected/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kube-api-access-rgtt9" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914450 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc64bc02-50f7-4594-8eaa-3a4ec97763f1" volumeName="kubernetes.io/projected/cc64bc02-50f7-4594-8eaa-3a4ec97763f1-kube-api-access-6nc4l" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914463 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eed1b9f7-3769-46a0-b8ba-63eb8d17a952" volumeName="kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914474 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11da27b-1a3d-4fca-b78e-fe9cd3caa46b" volumeName="kubernetes.io/projected/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-kube-api-access-4rnmt" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914487 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f70a3801-ffa0-4049-aeee-0d886ccb30b3" volumeName="kubernetes.io/configmap/f70a3801-ffa0-4049-aeee-0d886ccb30b3-config" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914499 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f645ece-f5a0-478c-a0a8-a772c740a072" volumeName="kubernetes.io/projected/1f645ece-f5a0-478c-a0a8-a772c740a072-kube-api-access-s6ds8" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914511 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56168f93-9536-4108-a5ce-5677c95ecf0c" volumeName="kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914536 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a" volumeName="kubernetes.io/projected/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-kube-api-access-74697" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914549 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f645ece-f5a0-478c-a0a8-a772c740a072" volumeName="kubernetes.io/configmap/1f645ece-f5a0-478c-a0a8-a772c740a072-iptables-alerter-script" seLinuxMountContext="" Nov 03 13:04:26.914524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914561 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c984d62-74a3-4dab-9639-8fed18759ddc" volumeName="kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 13:04:26.916545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914573 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99ae820d-6217-425b-9451-247f5dff70dc" volumeName="kubernetes.io/projected/99ae820d-6217-425b-9451-247f5dff70dc-kube-api-access-l9htq" seLinuxMountContext="" Nov 03 13:04:26.916545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914586 4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc" volumeName="kubernetes.io/secret/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-agent-certs" seLinuxMountContext="" Nov 03 13:04:26.916545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914597 4924 reconstruct.go:97] "Volume reconstruction finished" Nov 03 13:04:26.916545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.914606 4924 reconciler.go:29] "Reconciler: start to sync state" Nov 03 13:04:26.918489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.918469 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/1.log" Nov 03 13:04:26.922815 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.922727 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/1.log" Nov 03 13:04:26.940003 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.939966 4924 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 13:04:26.942696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.942674 4924 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 13:04:26.942696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.942698 4924 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 13:04:26.942851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.942723 4924 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 13:04:26.942851 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:26.942761 4924 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 13:04:26.961560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.961532 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/1.log" Nov 03 13:04:26.961704 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.961569 4924 generic.go:356] "Generic (PLEG): container finished" podID="50d1326a-e89d-4e45-90c6-9a254a157e48" containerID="17d482226f7b4a1122e534d47d3499d4d826d795c3594799547107c9df888788" exitCode=255 Nov 03 13:04:26.988634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.988551 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="b72b2e7663ee6c9e70f4747af575b5c3639ba5c71bd5859936628de28e58047a" exitCode=0 Nov 03 13:04:26.988634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.988587 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="af0d55183b8184a93e9f93cd1c203680f3e9fa12c7556bd29f350cbe462308e9" exitCode=0 Nov 03 13:04:26.988634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.988598 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="3a054f99789efdaca945d0668cc0d5bff33a7e19d70f2040d39101e5bec37312" exitCode=0 Nov 03 13:04:26.988634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.988605 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="28c64cd91bfb828140ef16887211d38e710476a20453e2d35c232e5c43526658" exitCode=0 Nov 03 13:04:26.988634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.988614 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="5ac0adfadcfa82314a6ceec07ba61355eef0c67b440caa0014af0cab44266eb7" exitCode=0 Nov 03 13:04:26.988634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:26.988621 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c984d62-74a3-4dab-9639-8fed18759ddc" containerID="7d3f9925cc596e905c0be838f59e0a31ac804d5ae072fc9d28fff169578b2cf2" exitCode=0 Nov 03 13:04:27.009901 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:27.009872 4924 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:04:27.017461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.017428 4924 generic.go:356] "Generic (PLEG): container finished" podID="f0d4fd5bce6fe0026d1ea2d7ee6f2f03" containerID="2fd266aa8699820e818da5056bb5b95f5edae66484b63281f1eb2a81fd89b9e1" exitCode=0 Nov 03 13:04:27.018906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.018885 4924 manager.go:324] Recovery completed Nov 03 13:04:27.037984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.037963 4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:27.039093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.039056 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:27.039188 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.039100 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:27.039188 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.039116 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:27.041278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041264 4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:27.041343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041265 4924 cpu_manager.go:226] "Starting" policy="none" Nov 03 13:04:27.041343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041337 4924 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 13:04:27.041406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041359 4924 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 13:04:27.041513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041498 4924 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 13:04:27.041564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041513 4924 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 13:04:27.041564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041535 4924 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 13:04:27.041564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041543 4924 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 13:04:27.041564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041550 4924 policy_none.go:50] "Start" Nov 03 13:04:27.041564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041560 4924 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 13:04:27.041685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041571 4924 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 13:04:27.041718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041690 4924 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 13:04:27.041718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041698 4924 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 13:04:27.041718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.041704 4924 policy_none.go:44] "Start" Nov 03 13:04:27.042085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.042057 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:27.042156 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.042105 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:27.042156 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.042120 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:27.042978 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:27.042963 4924 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 13:04:27.046134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.046107 4924 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 13:04:27.049404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.049391 4924 manager.go:341] "Starting Device Plugin manager" Nov 03 13:04:27.049466 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:27.049415 4924 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 13:04:27.049466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.049424 4924 server.go:87] "Starting device plugin registration server" Nov 03 13:04:27.049728 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.049716 4924 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 13:04:27.049759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.049730 4924 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 13:04:27.049836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.049818 4924 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 13:04:27.049969 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.049928 4924 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 13:04:27.050088 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.049975 4924 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 13:04:27.050088 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.049981 4924 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 13:04:27.050853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.050021 4924 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:04:27.04996339Z","UUID":"1334a74a-424c-45f4-96e1-fe9c2e6dadf0","Handler":null,"Name":"","Endpoint":""} Nov 03 13:04:27.051210 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:27.051183 4924 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 13:04:27.051286 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:27.051247 4924 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-63.us-west-2.compute.internal\" not found" Nov 03 13:04:27.052087 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.052052 4924 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 13:04:27.052167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.052094 4924 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 13:04:27.077836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.077811 4924 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:27.077967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.077951 4924 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 13:04:27.081504 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.081485 4924 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:27.123825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.123803 4924 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:27.129139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.129122 4924 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:27.150533 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.150505 4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:27.151542 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.151522 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:27.151625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.151555 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:27.151625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.151568 4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-63.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:27.151625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.151618 4924 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.244099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.243995 4924 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal"] Nov 03 13:04:27.244731 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.244711 4924 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.244790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.244729 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"f0d4fd5bce6fe0026d1ea2d7ee6f2f03","Type":"ContainerStarted","Data":"99b8db5f8d4c2c2dfafcc4174e78edace193eed84a541d7d2e271d603721b4b8"} Nov 03 13:04:27.244790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.244772 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"f0d4fd5bce6fe0026d1ea2d7ee6f2f03","Type":"ContainerDied","Data":"2fd266aa8699820e818da5056bb5b95f5edae66484b63281f1eb2a81fd89b9e1"} Nov 03 13:04:27.244879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.244794 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"f0d4fd5bce6fe0026d1ea2d7ee6f2f03","Type":"ContainerStarted","Data":"cc857669bc04ba5000b5a232dc118f91b3248463f19cff875d13c7711cc6844d"} Nov 03 13:04:27.244879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.244813 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"8df29524ca054c489efb3ae39305cb76","Type":"ContainerStarted","Data":"2b5f28c8e66cb821be25f88faef4781dde629c3f77d1ffffcb48e7a248bfdc85"} Nov 03 13:04:27.244879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.244832 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" event={"ID":"8df29524ca054c489efb3ae39305cb76","Type":"ContainerStarted","Data":"083d68b9024a7349f0bfe6d845b4d847dfb1d732c4910ebe1bca351cdf28db04"} Nov 03 13:04:27.245035 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.244957 4924 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.253575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.253551 4924 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.253662 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.253616 4924 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.302755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.302722 4924 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:27.302908 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:27.302771 4924 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.303250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.303237 4924 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:27.303298 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:27.303272 4924 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.379788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.379748 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.379788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.379790 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.379955 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.379806 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df29524ca054c489efb3ae39305cb76-config\") pod \"kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"8df29524ca054c489efb3ae39305cb76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.480523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.480491 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.480523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.480520 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.480694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.480541 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.480694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.480580 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d4fd5bce6fe0026d1ea2d7ee6f2f03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"f0d4fd5bce6fe0026d1ea2d7ee6f2f03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.480694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.480580 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df29524ca054c489efb3ae39305cb76-config\") pod \"kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"8df29524ca054c489efb3ae39305cb76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.480694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.480610 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8df29524ca054c489efb3ae39305cb76-config\") pod \"kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal\" (UID: \"8df29524ca054c489efb3ae39305cb76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:27.884761 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:27.884731 4924 apiserver.go:52] "Watching apiserver" Nov 03 13:04:28.024195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.024165 4924 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:28.024357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.024325 4924 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:28.074783 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.074755 4924 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 13:04:28.075627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.075601 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8478467d49-l5dtm","openshift-insights/insights-operator-7c4986d449-xhtjf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-nh6c9","openshift-dns/node-resolver-97s5t","openshift-image-registry/image-registry-896bc546-4fp47","openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk","openshift-cluster-node-tuning-operator/tuned-v2pft","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs","openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g","kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-6ct64","kube-system/kas-connection-checker-685895c7c9-92c7q","openshift-multus/multus-additional-cni-plugins-l6d9s","openshift-multus/network-metrics-daemon-ctkjc","openshift-ovn-kubernetes/ovnkube-node-kf7m8","openshift-network-console/networking-console-plugin-86c478bb88-67jgd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j","openshift-console-operator/console-operator-7cd56b8dbc-2pfz7","openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv","openshift-insights/insights-runtime-extractor-cq7lf","openshift-ingress-canary/ingress-canary-hp7ng","openshift-multus/multus-t9zkj","kube-system/kas-connection-checker-685895c7c9-f6bgd","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w","openshift-image-registry/image-registry-5564c5bd47-g6p67","kube-system/konnectivity-agent-g6wbc","kube-system/kas-connection-checker-685895c7c9-w6sm7","openshift-image-registry/node-ca-q7gwd","openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr","openshift-network-operator/iptables-alerter-sdwbl","openshift-dns/dns-default-k6d8j"] Nov 03 13:04:28.079410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.079379 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:28.079932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.079912 4924 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:28.080010 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.079964 4924 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:28.080206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.080185 4924 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:28.080473 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.080455 4924 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-63.us-west-2.compute.internal" Nov 03 13:04:28.080667 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.080642 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.081006 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.080991 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:28.081383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.081367 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.082869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.082852 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:28.085485 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.085468 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:28.088958 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.088942 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.090248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.090234 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.091148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.091130 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.112000 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.111976 4924 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 13:04:28.125755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.125726 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.125755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.125746 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.130327 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130304 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.130327 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130315 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 13:04:28.130464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130334 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 13:04:28.130641 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130615 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.130821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130662 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 13:04:28.130821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130690 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.130821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130762 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.130821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130783 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 13:04:28.130821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130784 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4m6js\"" Nov 03 13:04:28.130821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130784 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.130821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130809 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130844 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130874 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.130959 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131059 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dh6vm\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131145 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131156 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131160 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131174 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131199 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131226 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-697z8\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131306 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131439 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131572 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7pkcc\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131579 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5prrc\"" Nov 03 13:04:28.131732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.131624 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.178684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.178651 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 13:04:28.179120 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.179027 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.179643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.179623 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 13:04:28.179883 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.179865 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 13:04:28.180580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.180556 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.180819 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.180799 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9xv7j\"" Nov 03 13:04:28.180874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.180834 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 13:04:28.180918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.180893 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.181085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.181053 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pvzn\"" Nov 03 13:04:28.181137 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.181108 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 13:04:28.181179 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.181149 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-85kzz\"" Nov 03 13:04:28.181425 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.181410 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m8wc5\"" Nov 03 13:04:28.181536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.181517 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5wxnl\"" Nov 03 13:04:28.182167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.182142 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 13:04:28.182281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.182167 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.182349 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.182328 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.182837 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.182806 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 13:04:28.183288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183271 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.183374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183342 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 13:04:28.183429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183345 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 13:04:28.183429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183377 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 13:04:28.183429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183373 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.183553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183380 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:04:28.183553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183473 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 13:04:28.183553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183499 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 13:04:28.183553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183507 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8p2l7\"" Nov 03 13:04:28.183553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183521 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 13:04:28.183759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183560 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.183759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183608 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 13:04:28.183759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183603 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 13:04:28.183759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183570 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.183759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183667 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.183759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183625 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.183925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.183877 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.184245 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184227 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g4lr6\"" Nov 03 13:04:28.184311 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184291 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nbm\"" Nov 03 13:04:28.184390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184298 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 13:04:28.184390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184303 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 13:04:28.184390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184305 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 13:04:28.184390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184351 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bsb9h\"" Nov 03 13:04:28.184390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184375 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.184390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184378 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184397 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184353 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184438 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184434 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-sctm8\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184478 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t56mb\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184519 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184519 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184566 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 13:04:28.184618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184603 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5ntlc\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184633 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184649 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pvm6f\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184652 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184683 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184737 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-brvx7\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184764 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184768 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184818 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184823 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8v28q\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184832 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184857 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 13:04:28.184950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184920 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s6pph\"" Nov 03 13:04:28.185424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.184984 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 13:04:28.185424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185059 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.185424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185183 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnhgg\"" Nov 03 13:04:28.185424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185188 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 13:04:28.185424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185363 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 13:04:28.185424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185374 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lbs9x\"" Nov 03 13:04:28.185424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185388 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l9fl5\"" Nov 03 13:04:28.185424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185419 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zf7\" (UniqueName: \"kubernetes.io/projected/7d1d0d87-bbca-4424-bfad-d0b312732700-kube-api-access-s2zf7\") pod \"network-check-source-6f78d59f99-4nxqr\" (UID: \"7d1d0d87-bbca-4424-bfad-d0b312732700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185450 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185477 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185489 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185428 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185502 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9jvp2\"" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185520 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185518 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185584 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185637 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185647 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fg5qt\"" Nov 03 13:04:28.185691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185663 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btxph\" (UniqueName: \"kubernetes.io/projected/249644e9-11b8-48ad-8e70-79e3b87c550a-kube-api-access-btxph\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:04:28.186008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185697 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-var-lib-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.186008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185771 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-node-log\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.186008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185804 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:28.186008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185847 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6ds8\" (UniqueName: \"kubernetes.io/projected/1f645ece-f5a0-478c-a0a8-a772c740a072-kube-api-access-s6ds8\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:04:28.186008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185976 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rnmt\" (UniqueName: \"kubernetes.io/projected/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-kube-api-access-4rnmt\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.186008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.185999 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79g67\" (UniqueName: \"kubernetes.io/projected/50d1326a-e89d-4e45-90c6-9a254a157e48-kube-api-access-79g67\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:28.186217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186015 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-cni-binary-copy\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.186217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186033 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-etc-kubernetes\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.186217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186050 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-tuned\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.186217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186081 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.186364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186225 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-stats-auth\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.186364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186247 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-tuned\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.186364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186254 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50d1326a-e89d-4e45-90c6-9a254a157e48-serving-cert\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:28.186364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186296 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:28.186364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186321 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-systemd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.186540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186352 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-ovn\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.186540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186409 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:28.186540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186434 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50d1326a-e89d-4e45-90c6-9a254a157e48-serving-cert\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:28.186540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186450 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-cni-binary-copy\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.186540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186473 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.186540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186498 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-stats-auth\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.186540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186534 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186561 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7h6\" (UniqueName: \"kubernetes.io/projected/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-api-access-jw7h6\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186561 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186577 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-host\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186599 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-systemd\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186606 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-original-pull-secret\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186633 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-nginx-conf\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186647 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qc5\" (UniqueName: \"kubernetes.io/projected/71f66428-6b35-45cc-ac46-ce5c07aa958c-kube-api-access-67qc5\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186670 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-serviceca\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186693 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186708 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186722 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f645ece-f5a0-478c-a0a8-a772c740a072-host-slash\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186746 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-socket-dir-parent\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.186772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186772 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-netns\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186783 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186795 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-kubernetes\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186841 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186875 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-serviceca\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.186989 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfz2t\" (UniqueName: \"kubernetes.io/projected/541aac48-8aa9-40e2-88ec-17687fdd5167-kube-api-access-rfz2t\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187023 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-multus\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187040 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187046 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-script-lib\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187063 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-cnibin\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.187201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187180 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187217 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249644e9-11b8-48ad-8e70-79e3b87c550a-tmp-dir\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187245 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-kubelet\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187271 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovn-node-metrics-cert\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187289 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-script-lib\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187299 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187302 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249644e9-11b8-48ad-8e70-79e3b87c550a-tmp-dir\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187330 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-kubelet\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187355 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187378 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-system-cni-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187403 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzwn\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187444 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovn-node-metrics-cert\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187454 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f70a3801-ffa0-4049-aeee-0d886ccb30b3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187507 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-k8s-cni-cncf-io\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187505 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187553 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187579 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-var-lib-kubelet\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187597 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thp2p\" (UniqueName: \"kubernetes.io/projected/dd77bb7b-1f79-4021-a9ad-70160e72cda4-kube-api-access-thp2p\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187598 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187615 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f70a3801-ffa0-4049-aeee-0d886ccb30b3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:28.187618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187613 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74697\" (UniqueName: \"kubernetes.io/projected/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-kube-api-access-74697\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187716 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187750 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/541aac48-8aa9-40e2-88ec-17687fdd5167-cert\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187774 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/f70a3801-ffa0-4049-aeee-0d886ccb30b3-kube-api-access-dl2z6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187803 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-registration-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187941 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccn5\" (UniqueName: \"kubernetes.io/projected/ec0b3420-306f-4a10-95ff-0bcf4a4f1730-kube-api-access-hccn5\") pod \"volume-data-source-validator-58c9f5f46d-x6jrs\" (UID: \"ec0b3420-306f-4a10-95ff-0bcf4a4f1730\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.187974 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71f66428-6b35-45cc-ac46-ce5c07aa958c-serving-cert\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188008 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/541aac48-8aa9-40e2-88ec-17687fdd5167-cert\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188026 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f645ece-f5a0-478c-a0a8-a772c740a072-iptables-alerter-script\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188046 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nc4l\" (UniqueName: \"kubernetes.io/projected/cc64bc02-50f7-4594-8eaa-3a4ec97763f1-kube-api-access-6nc4l\") pod \"migrator-66df5d496b-82p5g\" (UID: \"cc64bc02-50f7-4594-8eaa-3a4ec97763f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188064 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-hostroot\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188105 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-netd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188134 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188145 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71f66428-6b35-45cc-ac46-ce5c07aa958c-serving-cert\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188178 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8425f3a6-d496-41eb-bb83-c65f6011890b-crio-socket\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188210 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188258 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188287 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f645ece-f5a0-478c-a0a8-a772c740a072-iptables-alerter-script\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188290 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188335 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae820d-6217-425b-9451-247f5dff70dc-tmp-dir\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188361 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-system-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188380 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188386 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysconfig\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188421 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99ae820d-6217-425b-9451-247f5dff70dc-tmp-dir\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188429 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188474 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9htq\" (UniqueName: \"kubernetes.io/projected/99ae820d-6217-425b-9451-247f5dff70dc-kube-api-access-l9htq\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.188527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188528 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f70a3801-ffa0-4049-aeee-0d886ccb30b3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188633 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188634 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-conf\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188673 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188694 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-env-overrides\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188723 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f70a3801-ffa0-4049-aeee-0d886ccb30b3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188738 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q822s\" (UniqueName: \"kubernetes.io/projected/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-kube-api-access-q822s\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188768 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188793 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-socket-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188853 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-env-overrides\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188903 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-kubelet-config\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188938 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188969 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.188976 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c984d62-74a3-4dab-9639-8fed18759ddc-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189011 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189038 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189119 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfds\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189149 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjwcb\" (UniqueName: \"kubernetes.io/projected/53c4e88e-1203-4083-98e9-9a2202a3a704-kube-api-access-kjwcb\") pod \"kas-connection-checker-685895c7c9-92c7q\" (UID: \"53c4e88e-1203-4083-98e9-9a2202a3a704\") " pod="kube-system/kas-connection-checker-685895c7c9-92c7q" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189170 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189176 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-agent-certs\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189206 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-dbus\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189232 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-konnectivity-ca\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189260 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189232 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189358 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-modprobe-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189380 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbch\" (UniqueName: \"kubernetes.io/projected/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-kube-api-access-cvbch\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189423 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-tmp\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.189441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189440 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-konnectivity-ca\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189475 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189493 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189491 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc-agent-certs\") pod \"konnectivity-agent-g6wbc\" (UID: \"c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc\") " pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189508 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-sys\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189529 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189562 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-etc-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189598 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189616 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-tmp\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189642 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-sys-fs\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189675 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-bin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189693 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-netns\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189727 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99ae820d-6217-425b-9451-247f5dff70dc-config-volume\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189769 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189853 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-serving-cert\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189880 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7fc\" (UniqueName: \"kubernetes.io/projected/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-kube-api-access-bp7fc\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189888 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99ae820d-6217-425b-9451-247f5dff70dc-config-volume\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189894 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189909 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71f66428-6b35-45cc-ac46-ce5c07aa958c-config\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189924 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-config\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189941 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-daemon-config\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.189978 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqn2s\" (UniqueName: \"kubernetes.io/projected/08ac07c7-d646-4d10-8ba3-844c752e3a37-kube-api-access-xqn2s\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190002 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-tmp\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190018 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-slash\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190042 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trds9\" (UniqueName: \"kubernetes.io/projected/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70-kube-api-access-trds9\") pod \"kas-connection-checker-685895c7c9-f6bgd\" (UID: \"78248e2d-8c0b-47fb-b55b-07e0ffdc1c70\") " pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190093 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-device-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190111 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-tmp\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190122 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-serving-cert\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190137 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-config\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190164 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190220 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-daemon-config\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190227 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-snapshots\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190269 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190259 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71f66428-6b35-45cc-ac46-ce5c07aa958c-config\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190309 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190317 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-snapshots\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190324 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszmj\" (UniqueName: \"kubernetes.io/projected/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-kube-api-access-gszmj\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190362 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-service-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190359 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190395 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-host\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190431 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190456 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-systemd-units\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.190526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190483 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-bin\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190559 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190586 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-default-certificate\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190611 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190640 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190755 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-trusted-ca\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190784 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtt9\" (UniqueName: \"kubernetes.io/projected/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kube-api-access-rgtt9\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190800 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-default-certificate\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190846 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8425f3a6-d496-41eb-bb83-c65f6011890b-data-volume\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190883 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-conf-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190906 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-run\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190945 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-config\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.190974 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191036 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8425f3a6-d496-41eb-bb83-c65f6011890b-data-volume\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191084 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191132 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdd6\" (UniqueName: \"kubernetes.io/projected/6c984d62-74a3-4dab-9639-8fed18759ddc-kube-api-access-lgdd6\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191162 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191222 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2289v\" (UniqueName: \"kubernetes.io/projected/6365006c-9e6f-406f-9102-e3b0b452675d-kube-api-access-2289v\") pod \"kas-connection-checker-685895c7c9-w6sm7\" (UID: \"6365006c-9e6f-406f-9102-e3b0b452675d\") " pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191266 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfgn\" (UniqueName: \"kubernetes.io/projected/05516dd7-e72f-4b64-9707-e401c7e6198b-kube-api-access-cpfgn\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191293 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249644e9-11b8-48ad-8e70-79e3b87c550a-hosts-file\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191315 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-cnibin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191322 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191339 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-multus-certs\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191519 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dd77bb7b-1f79-4021-a9ad-70160e72cda4-ovnkube-config\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191564 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-lib-modules\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.191661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.191591 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-log-socket\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.219300 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.219272 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 13:04:28.224280 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.224262 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 13:04:28.226471 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.226453 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:28.231737 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.231719 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50d1326a-e89d-4e45-90c6-9a254a157e48-trusted-ca\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:28.237722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.237691 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6ds8\" (UniqueName: \"kubernetes.io/projected/1f645ece-f5a0-478c-a0a8-a772c740a072-kube-api-access-s6ds8\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:04:28.237840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.237783 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btxph\" (UniqueName: \"kubernetes.io/projected/249644e9-11b8-48ad-8e70-79e3b87c550a-kube-api-access-btxph\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:04:28.238009 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.237986 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zf7\" (UniqueName: \"kubernetes.io/projected/7d1d0d87-bbca-4424-bfad-d0b312732700-kube-api-access-s2zf7\") pod \"network-check-source-6f78d59f99-4nxqr\" (UID: \"7d1d0d87-bbca-4424-bfad-d0b312732700\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4nxqr" Nov 03 13:04:28.238376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.238353 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79g67\" (UniqueName: \"kubernetes.io/projected/50d1326a-e89d-4e45-90c6-9a254a157e48-kube-api-access-79g67\") pod \"console-operator-7cd56b8dbc-2pfz7\" (UID: \"50d1326a-e89d-4e45-90c6-9a254a157e48\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:28.238563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.238541 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rnmt\" (UniqueName: \"kubernetes.io/projected/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-kube-api-access-4rnmt\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.238563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.238550 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.238947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.238924 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7h6\" (UniqueName: \"kubernetes.io/projected/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-api-access-jw7h6\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.239018 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.238925 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qc5\" (UniqueName: \"kubernetes.io/projected/71f66428-6b35-45cc-ac46-ce5c07aa958c-kube-api-access-67qc5\") pod \"service-ca-operator-544d749c8c-zphsk\" (UID: \"71f66428-6b35-45cc-ac46-ce5c07aa958c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" Nov 03 13:04:28.239018 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.238991 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfz2t\" (UniqueName: \"kubernetes.io/projected/541aac48-8aa9-40e2-88ec-17687fdd5167-kube-api-access-rfz2t\") pod \"ingress-canary-hp7ng\" (UID: \"541aac48-8aa9-40e2-88ec-17687fdd5167\") " pod="openshift-ingress-canary/ingress-canary-hp7ng" Nov 03 13:04:28.239342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.239110 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzwn\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.239342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.239261 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thp2p\" (UniqueName: \"kubernetes.io/projected/dd77bb7b-1f79-4021-a9ad-70160e72cda4-kube-api-access-thp2p\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.274817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.274773 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nc4l\" (UniqueName: \"kubernetes.io/projected/cc64bc02-50f7-4594-8eaa-3a4ec97763f1-kube-api-access-6nc4l\") pod \"migrator-66df5d496b-82p5g\" (UID: \"cc64bc02-50f7-4594-8eaa-3a4ec97763f1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" Nov 03 13:04:28.274973 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.274930 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9htq\" (UniqueName: \"kubernetes.io/projected/99ae820d-6217-425b-9451-247f5dff70dc-kube-api-access-l9htq\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.275037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.274949 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbch\" (UniqueName: \"kubernetes.io/projected/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-kube-api-access-cvbch\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.275258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.275230 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74697\" (UniqueName: \"kubernetes.io/projected/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-kube-api-access-74697\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:28.275366 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.275351 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q822s\" (UniqueName: \"kubernetes.io/projected/52b2bd33-97be-42ee-b7d4-2b86f1de3d7d-kube-api-access-q822s\") pod \"insights-operator-7c4986d449-xhtjf\" (UID: \"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d\") " pod="openshift-insights/insights-operator-7c4986d449-xhtjf" Nov 03 13:04:28.276308 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.276244 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszmj\" (UniqueName: \"kubernetes.io/projected/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-kube-api-access-gszmj\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:28.276439 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.276421 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccn5\" (UniqueName: \"kubernetes.io/projected/ec0b3420-306f-4a10-95ff-0bcf4a4f1730-kube-api-access-hccn5\") pod \"volume-data-source-validator-58c9f5f46d-x6jrs\" (UID: \"ec0b3420-306f-4a10-95ff-0bcf4a4f1730\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-x6jrs" Nov 03 13:04:28.276512 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.276493 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqn2s\" (UniqueName: \"kubernetes.io/projected/08ac07c7-d646-4d10-8ba3-844c752e3a37-kube-api-access-xqn2s\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.276907 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.276893 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfds\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.277677 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.277644 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/f70a3801-ffa0-4049-aeee-0d886ccb30b3-kube-api-access-dl2z6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg68w\" (UID: \"f70a3801-ffa0-4049-aeee-0d886ccb30b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" Nov 03 13:04:28.277988 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.277959 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7fc\" (UniqueName: \"kubernetes.io/projected/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-kube-api-access-bp7fc\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:04:28.278153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.278106 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2289v\" (UniqueName: \"kubernetes.io/projected/6365006c-9e6f-406f-9102-e3b0b452675d-kube-api-access-2289v\") pod \"kas-connection-checker-685895c7c9-w6sm7\" (UID: \"6365006c-9e6f-406f-9102-e3b0b452675d\") " pod="kube-system/kas-connection-checker-685895c7c9-w6sm7" Nov 03 13:04:28.278739 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.278720 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbkm\" (UniqueName: \"kubernetes.io/projected/1651b048-50ba-4391-8382-60f9cffd1fa7-kube-api-access-mhbkm\") pod \"network-check-target-nh6c9\" (UID: \"1651b048-50ba-4391-8382-60f9cffd1fa7\") " pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:28.279383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.279365 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfgn\" (UniqueName: \"kubernetes.io/projected/05516dd7-e72f-4b64-9707-e401c7e6198b-kube-api-access-cpfgn\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:28.279427 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.279401 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjwcb\" (UniqueName: \"kubernetes.io/projected/53c4e88e-1203-4083-98e9-9a2202a3a704-kube-api-access-kjwcb\") pod \"kas-connection-checker-685895c7c9-92c7q\" (UID: \"53c4e88e-1203-4083-98e9-9a2202a3a704\") " pod="kube-system/kas-connection-checker-685895c7c9-92c7q" Nov 03 13:04:28.280108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.280089 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.280108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.280098 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtt9\" (UniqueName: \"kubernetes.io/projected/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kube-api-access-rgtt9\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.280590 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.280571 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trds9\" (UniqueName: \"kubernetes.io/projected/78248e2d-8c0b-47fb-b55b-07e0ffdc1c70-kube-api-access-trds9\") pod \"kas-connection-checker-685895c7c9-f6bgd\" (UID: \"78248e2d-8c0b-47fb-b55b-07e0ffdc1c70\") " pod="kube-system/kas-connection-checker-685895c7c9-f6bgd" Nov 03 13:04:28.280840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.280826 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdd6\" (UniqueName: \"kubernetes.io/projected/6c984d62-74a3-4dab-9639-8fed18759ddc-kube-api-access-lgdd6\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.292427 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292404 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.292528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292433 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-registration-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.292528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292452 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-hostroot\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.292528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292467 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-netd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.292528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292510 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-hostroot\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.292528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292514 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8425f3a6-d496-41eb-bb83-c65f6011890b-crio-socket\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292526 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292533 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-registration-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292543 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292565 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292583 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292536 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-netd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292583 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8425f3a6-d496-41eb-bb83-c65f6011890b-crio-socket\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292603 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-system-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292632 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292642 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292636 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysconfig\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292662 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysconfig\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.292675 4924 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:28.292691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292685 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-system-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292677 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-conf\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.292762 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.792745636 +0000 UTC m=+1.987149212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292786 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292804 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-socket-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292765 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-conf\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292824 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-kubelet-config\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292840 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292850 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292873 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-socket-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292882 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-kubelet-config\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292884 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292877 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292920 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-dbus\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.292935 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292940 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.292945 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.292957 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-modprobe-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.292983 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.792970129 +0000 UTC m=+1.987373716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293000 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-sys\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293006 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eed1b9f7-3769-46a0-b8ba-63eb8d17a952-dbus\") pod \"global-pull-secret-syncer-6ct64\" (UID: \"eed1b9f7-3769-46a0-b8ba-63eb8d17a952\") " pod="kube-system/global-pull-secret-syncer-6ct64" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293025 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293051 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-etc-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293055 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-sys\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293064 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.793045482 +0000 UTC m=+1.987449070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:28.293126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293084 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-modprobe-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293146 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293171 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293189 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-sys-fs\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293208 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-bin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293224 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.79320982 +0000 UTC m=+1.987613405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293231 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-bin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293171 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-etc-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293256 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-netns\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293264 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-sys-fs\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293291 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-slash\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293316 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-device-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293339 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-run-netns\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293341 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293376 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293386 4924 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293385 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3-device-dir\") pod \"aws-ebs-csi-driver-node-fnz7j\" (UID: \"a3a6ef94-ada1-44d6-9dc4-33a3c940fcb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fnz7j" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293346 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-slash\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293400 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-host\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293424 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.793410703 +0000 UTC m=+1.987814291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : configmap "kube-rbac-proxy" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293452 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-systemd-units\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293456 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4b7eaba1-903d-4c79-9cb9-f0ba75d8b970-host\") pod \"node-ca-q7gwd\" (UID: \"4b7eaba1-903d-4c79-9cb9-f0ba75d8b970\") " pod="openshift-image-registry/node-ca-q7gwd" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293450 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293477 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-bin\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293493 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-systemd-units\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293509 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293534 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293551 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-cni-bin\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293560 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-conf-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293587 4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293598 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-conf-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293620 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-run\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293589 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-run\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293642 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.793628333 +0000 UTC m=+1.988031914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293647 4924 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293679 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293685 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.793673907 +0000 UTC m=+1.988077496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293708 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293725 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249644e9-11b8-48ad-8e70-79e3b87c550a-hosts-file\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293739 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-cnibin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293743 4924 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293753 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-multus-certs\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293795 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.793782178 +0000 UTC m=+1.988185753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293794 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-multus-certs\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293803 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/249644e9-11b8-48ad-8e70-79e3b87c550a-hosts-file\") pod \"node-resolver-97s5t\" (UID: \"249644e9-11b8-48ad-8e70-79e3b87c550a\") " pod="openshift-dns/node-resolver-97s5t" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293809 4924 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293812 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-cnibin\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293807 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-lib-modules\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293840 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.793832624 +0000 UTC m=+1.988236228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : secret "metrics-daemon-secret" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293851 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-log-socket\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293875 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293906 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-lib-modules\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293909 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-log-socket\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293907 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293941 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-var-lib-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293957 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-node-log\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293963 4924 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293969 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-var-lib-openvswitch\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293941 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-cni-dir\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293984 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-etc-kubernetes\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.293990 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.793982133 +0000 UTC m=+1.988385721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.293998 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-node-log\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294010 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-etc-kubernetes\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294035 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-systemd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294058 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-ovn\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294086 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-systemd\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294110 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-host\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294122 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-run-ovn\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294129 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-systemd\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294156 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-systemd\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294173 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f645ece-f5a0-478c-a0a8-a772c740a072-host-slash\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:04:28.294138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294189 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-host\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294201 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-socket-dir-parent\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294229 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-netns\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294243 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f645ece-f5a0-478c-a0a8-a772c740a072-host-slash\") pod \"iptables-alerter-sdwbl\" (UID: \"1f645ece-f5a0-478c-a0a8-a772c740a072\") " pod="openshift-network-operator/iptables-alerter-sdwbl" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294252 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-kubernetes\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294283 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-multus-socket-dir-parent\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294274 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-netns\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294284 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-multus\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294317 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-kubernetes\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294321 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-cni-multus\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294323 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-cnibin\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294348 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-cnibin\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294359 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-kubelet\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294386 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-kubelet\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294406 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd77bb7b-1f79-4021-a9ad-70160e72cda4-host-kubelet\") pod \"ovnkube-node-kf7m8\" (UID: \"dd77bb7b-1f79-4021-a9ad-70160e72cda4\") " pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294410 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294437 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-system-cni-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294457 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-k8s-cni-cncf-io\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294477 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-var-lib-kubelet\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294481 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294501 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c984d62-74a3-4dab-9639-8fed18759ddc-system-cni-dir\") pod \"multus-additional-cni-plugins-l6d9s\" (UID: \"6c984d62-74a3-4dab-9639-8fed18759ddc\") " pod="openshift-multus/multus-additional-cni-plugins-l6d9s" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294504 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08ac07c7-d646-4d10-8ba3-844c752e3a37-host-run-k8s-cni-cncf-io\") pod \"multus-t9zkj\" (UID: \"08ac07c7-d646-4d10-8ba3-844c752e3a37\") " pod="openshift-multus/multus-t9zkj" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294513 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-var-lib-kubelet\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.294531 4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294551 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-var-lib-kubelet\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.294559 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:28.794551457 +0000 UTC m=+1.988955066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:28.296186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.294530 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d90132-40a2-4f02-8f59-e0059a1c0a1a-etc-sysctl-d\") pod \"tuned-v2pft\" (UID: \"d8d90132-40a2-4f02-8f59-e0059a1c0a1a\") " pod="openshift-cluster-node-tuning-operator/tuned-v2pft" Nov 03 13:04:28.382775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.382735 4924 scope.go:122] "RemoveContainer" containerID="17d482226f7b4a1122e534d47d3499d4d826d795c3594799547107c9df888788" Nov 03 13:04:28.393038 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.393018 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:04:28.798536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798439 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:28.798536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798502 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798547 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798577 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798589 4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798603 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798640 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798668 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798653413 +0000 UTC m=+2.993056990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798674 4924 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798700 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798682777 +0000 UTC m=+2.993086366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798731 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798719042 +0000 UTC m=+2.993122628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798732 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798752 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798753 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798775 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798788 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798787 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:28.798798 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798798 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798786871 +0000 UTC m=+2.993190464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798821 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798809699 +0000 UTC m=+2.993213289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798844 4924 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798865 4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798866 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798874 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798864827 +0000 UTC m=+2.993268416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798897 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798888896 +0000 UTC m=+2.993292472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798912 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798920 4924 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.798938 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798943 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798936766 +0000 UTC m=+2.993340377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.798980 4924 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.799001 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.798993984 +0000 UTC m=+2.993397556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.799012 4924 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:28.799049 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:04:29.799039133 +0000 UTC m=+2.993442706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : secret "metrics-daemon-secret" not found Nov 03 13:04:28.799497 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.799238 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8425f3a6-d496-41eb-bb83-c65f6011890b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:28.810744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:28.810682 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:29.028785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.028760 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:04:29.029218 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.029189 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/1.log" Nov 03 13:04:29.029262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.029221 4924 generic.go:356] "Generic (PLEG): container finished" podID="50d1326a-e89d-4e45-90c6-9a254a157e48" containerID="ef431b7ea3ee1e21ad2dc173c670342bbc798c5406dcf3f93793556df981e3f1" exitCode=255 Nov 03 13:04:29.029324 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.029303 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" event={"ID":"50d1326a-e89d-4e45-90c6-9a254a157e48","Type":"ContainerDied","Data":"ef431b7ea3ee1e21ad2dc173c670342bbc798c5406dcf3f93793556df981e3f1"} Nov 03 13:04:29.029371 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.029363 4924 scope.go:122] "RemoveContainer" containerID="17d482226f7b4a1122e534d47d3499d4d826d795c3594799547107c9df888788" Nov 03 13:04:29.029889 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.029864 4924 scope.go:122] "RemoveContainer" containerID="ef431b7ea3ee1e21ad2dc173c670342bbc798c5406dcf3f93793556df981e3f1" Nov 03 13:04:29.030060 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.030041 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2pfz7_openshift-console-operator(50d1326a-e89d-4e45-90c6-9a254a157e48)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" podUID="50d1326a-e89d-4e45-90c6-9a254a157e48" Nov 03 13:04:29.161760 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.161714 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:29.808405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808315 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:29.808405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808365 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:29.808405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808392 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808413 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808432 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808478 4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808510 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808521 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808546 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808531044 +0000 UTC m=+5.002934616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808560 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808554282 +0000 UTC m=+5.002957854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808565 4924 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808571 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808565322 +0000 UTC m=+5.002968894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808602 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808630 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808631 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:29.808677 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808671 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808655924 +0000 UTC m=+5.003059512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808689 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808679476 +0000 UTC m=+5.003083055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808708 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808748 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808712 4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808777 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808803 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808792366 +0000 UTC m=+5.003195955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808763 4924 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:29.808829 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808833 4924 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808841 4924 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808852 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808835339 +0000 UTC m=+5.003238926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808866 4924 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808876 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808865812 +0000 UTC m=+5.003269398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808898 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808889686 +0000 UTC m=+5.003293259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : secret "metrics-daemon-secret" not found Nov 03 13:04:29.809186 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:29.808914 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.808906613 +0000 UTC m=+5.003310191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:30.034231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:30.034204 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:04:30.034661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:30.034589 4924 scope.go:122] "RemoveContainer" containerID="ef431b7ea3ee1e21ad2dc173c670342bbc798c5406dcf3f93793556df981e3f1" Nov 03 13:04:30.034838 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:30.034814 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2pfz7_openshift-console-operator(50d1326a-e89d-4e45-90c6-9a254a157e48)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" podUID="50d1326a-e89d-4e45-90c6-9a254a157e48" Nov 03 13:04:30.035802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:30.035780 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-82p5g" event={"ID":"cc64bc02-50f7-4594-8eaa-3a4ec97763f1","Type":"ContainerStarted","Data":"d9234859985908d16ad3a55779f7ac378d7b33b2b4b997b29eb59a9d3682fafb"} Nov 03 13:04:31.250907 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.250876 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:31.473403 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.473373 4924 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:31.473806 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.473790 4924 scope.go:122] "RemoveContainer" containerID="ef431b7ea3ee1e21ad2dc173c670342bbc798c5406dcf3f93793556df981e3f1" Nov 03 13:04:31.474014 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.473992 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2pfz7_openshift-console-operator(50d1326a-e89d-4e45-90c6-9a254a157e48)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" podUID="50d1326a-e89d-4e45-90c6-9a254a157e48" Nov 03 13:04:31.495639 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.495614 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-97s5t_249644e9-11b8-48ad-8e70-79e3b87c550a/dns-node-resolver/0.log" Nov 03 13:04:31.831155 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831119 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831164 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831191 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831219 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831242 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831276 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831280 4924 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831281 4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831358 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831372 4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831374 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:31.831386 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831354 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.831334774 +0000 UTC m=+9.025738363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831409 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.831393503 +0000 UTC m=+9.025797085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : configmap references non-existent config key: service-ca.crt Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831422 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls podName:278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.831415492 +0000 UTC m=+9.025819064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6cpnv" (UID: "278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831429 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831443 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831434 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.831427553 +0000 UTC m=+9.025831125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831481 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.831472282 +0000 UTC m=+9.025875859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831494 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831518 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831537 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.831524028 +0000 UTC m=+9.025927609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831556 4924 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831567 4924 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831580 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831577 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert podName:f4f67336-02d0-46a8-9389-e4a1e2a80ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.831571568 +0000 UTC m=+9.025975192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-67jgd" (UID: "f4f67336-02d0-46a8-9389-e4a1e2a80ca6") : secret "networking-console-plugin-cert" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831636 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs podName:f11da27b-1a3d-4fca-b78e-fe9cd3caa46b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.83162171 +0000 UTC m=+9.026025288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs") pod "router-default-8478467d49-l5dtm" (UID: "f11da27b-1a3d-4fca-b78e-fe9cd3caa46b") : secret "router-metrics-certs-default" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831646 4924 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:31.831679 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831684 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.831673519 +0000 UTC m=+9.026077106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : secret "metrics-daemon-secret" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831772 4924 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:31.831859 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:31.831807 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls podName:99ae820d-6217-425b-9451-247f5dff70dc nodeName:}" failed. No retries permitted until 2026-11-03 13:04:35.83180004 +0000 UTC m=+9.026203612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls") pod "dns-default-k6d8j" (UID: "99ae820d-6217-425b-9451-247f5dff70dc") : secret "dns-default-metrics-tls" not found Nov 03 13:04:32.297782 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:32.297754 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q7gwd_4b7eaba1-903d-4c79-9cb9-f0ba75d8b970/node-ca/0.log" Nov 03 13:04:32.599467 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:32.599384 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:32.951220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:32.951185 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:33.062508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:33.062468 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:33.099161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:33.099139 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hp7ng_541aac48-8aa9-40e2-88ec-17687fdd5167/serve-healthcheck-canary/0.log" Nov 03 13:04:33.176372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:33.176339 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:33.695372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:33.695341 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-82p5g_cc64bc02-50f7-4594-8eaa-3a4ec97763f1/migrator/0.log" Nov 03 13:04:33.718514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:33.718484 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:33.809507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:33.809466 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:33.896032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:33.895999 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-xg68w_f70a3801-ffa0-4049-aeee-0d886ccb30b3/kube-storage-version-migrator-operator/0.log" Nov 03 13:04:34.127251 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:34.127171 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:34.254642 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:34.254615 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:34.520148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:34.520114 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:34.684153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:34.684128 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:34.864638 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:34.864566 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:34.922087 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:34.922041 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:35.051221 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.051192 4924 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:04:35.320601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.320561 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:35.720485 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.720453 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:35.830920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.830883 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:35.870236 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870210 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870254 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870278 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870316 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870342 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870362 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870376 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870408 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870382 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5564c5bd47-g6p67: secret "image-registry-tls" not found Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870447 4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870394 4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870516 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls podName:8425f3a6-d496-41eb-bb83-c65f6011890b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:43.870498088 +0000 UTC m=+17.064901679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cq7lf" (UID: "8425f3a6-d496-41eb-bb83-c65f6011890b") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870448 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870525 4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896bc546-4fp47: secret "image-registry-tls" not found Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870538 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls podName:a8cd572f-aee9-4f80-87c2-2504a0ba404d nodeName:}" failed. No retries permitted until 2026-11-03 13:04:43.870526095 +0000 UTC m=+17.064929689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls") pod "image-registry-5564c5bd47-g6p67" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d") : secret "image-registry-tls" not found Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870613 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls podName:56168f93-9536-4108-a5ce-5677c95ecf0c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:43.870591162 +0000 UTC m=+17.064994737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls") pod "image-registry-896bc546-4fp47" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c") : secret "image-registry-tls" not found Nov 03 13:04:35.870649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870644 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:35.871343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870696 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:35.871343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.870731 4924 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 13:04:35.871343 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870818 4924 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:04:35.871343 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870859 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls podName:05516dd7-e72f-4b64-9707-e401c7e6198b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:43.870846958 +0000 UTC m=+17.065250530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r2ftp" (UID: "05516dd7-e72f-4b64-9707-e401c7e6198b") : secret "samples-operator-tls" not found Nov 03 13:04:35.871343 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.870940 4924 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:35.871343 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:35.871003 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs podName:cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a nodeName:}" failed. No retries permitted until 2026-11-03 13:04:43.870988522 +0000 UTC m=+17.065392105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs") pod "network-metrics-daemon-ctkjc" (UID: "cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a") : secret "metrics-daemon-secret" not found Nov 03 13:04:35.871343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.871002 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-service-ca-bundle\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:35.872972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.872934 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99ae820d-6217-425b-9451-247f5dff70dc-metrics-tls\") pod \"dns-default-k6d8j\" (UID: \"99ae820d-6217-425b-9451-247f5dff70dc\") " pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:35.873131 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.873115 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6cpnv\" (UID: \"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:35.873298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.873281 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4f67336-02d0-46a8-9389-e4a1e2a80ca6-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-67jgd\" (UID: \"f4f67336-02d0-46a8-9389-e4a1e2a80ca6\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:35.873439 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.873420 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11da27b-1a3d-4fca-b78e-fe9cd3caa46b-metrics-certs\") pod \"router-default-8478467d49-l5dtm\" (UID: \"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b\") " pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:35.881482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.881464 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" Nov 03 13:04:35.882609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.882591 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:35.883132 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.883110 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" Nov 03 13:04:35.891184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:35.891161 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:36.119331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.119300 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:36.227351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.227306 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:36.264404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.264308 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv"] Nov 03 13:04:36.295716 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:36.295689 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278ee3e6_54bd_4ae0_8ce1_fbda1caad4b2.slice/crio-c5b51b8066eb9538000a5fa8ed9cb3b39b2be1a6b8fbad98f580a845380f2d05 WatchSource:0}: Error finding container c5b51b8066eb9538000a5fa8ed9cb3b39b2be1a6b8fbad98f580a845380f2d05: Status 404 returned error can't find the container with id c5b51b8066eb9538000a5fa8ed9cb3b39b2be1a6b8fbad98f580a845380f2d05 Nov 03 13:04:36.302821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.302798 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-67jgd"] Nov 03 13:04:36.305513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.305490 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8478467d49-l5dtm"] Nov 03 13:04:36.306027 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:36.306006 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f67336_02d0_46a8_9389_e4a1e2a80ca6.slice/crio-6ff7f7596e412942ed87aa422daac178ecea6e4ed92a5da2664b679a0947b5f5 WatchSource:0}: Error finding container 6ff7f7596e412942ed87aa422daac178ecea6e4ed92a5da2664b679a0947b5f5: Status 404 returned error can't find the container with id 6ff7f7596e412942ed87aa422daac178ecea6e4ed92a5da2664b679a0947b5f5 Nov 03 13:04:36.342034 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:36.342010 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11da27b_1a3d_4fca_b78e_fe9cd3caa46b.slice/crio-7d55ff04612be6c00c96b9aff5bb7dab2f2b2134b8a9340fab9045c9282a2193 WatchSource:0}: Error finding container 7d55ff04612be6c00c96b9aff5bb7dab2f2b2134b8a9340fab9045c9282a2193: Status 404 returned error can't find the container with id 7d55ff04612be6c00c96b9aff5bb7dab2f2b2134b8a9340fab9045c9282a2193 Nov 03 13:04:36.348562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.348538 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k6d8j"] Nov 03 13:04:36.352428 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:36.352407 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ae820d_6217_425b_9451_247f5dff70dc.slice/crio-5d90da68f72066680f91e55677c8b786fda2ea7821a2da89714a09c134c0a7cf WatchSource:0}: Error finding container 5d90da68f72066680f91e55677c8b786fda2ea7821a2da89714a09c134c0a7cf: Status 404 returned error can't find the container with id 5d90da68f72066680f91e55677c8b786fda2ea7821a2da89714a09c134c0a7cf Nov 03 13:04:36.463432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.463407 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kf7m8" Nov 03 13:04:36.518487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.518404 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:36.925123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:36.925097 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:37.038213 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.038187 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:37.057506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.057475 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6d8j" event={"ID":"99ae820d-6217-425b-9451-247f5dff70dc","Type":"ContainerStarted","Data":"5d90da68f72066680f91e55677c8b786fda2ea7821a2da89714a09c134c0a7cf"} Nov 03 13:04:37.059136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.059107 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8478467d49-l5dtm" event={"ID":"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b","Type":"ContainerStarted","Data":"b69b072e9c8776b6bff2a3afdf824be85deceb10947284d05ca7de7cb94cf957"} Nov 03 13:04:37.059264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.059153 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8478467d49-l5dtm" event={"ID":"f11da27b-1a3d-4fca-b78e-fe9cd3caa46b","Type":"ContainerStarted","Data":"7d55ff04612be6c00c96b9aff5bb7dab2f2b2134b8a9340fab9045c9282a2193"} Nov 03 13:04:37.060431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.060408 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" event={"ID":"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2","Type":"ContainerStarted","Data":"c5b51b8066eb9538000a5fa8ed9cb3b39b2be1a6b8fbad98f580a845380f2d05"} Nov 03 13:04:37.061484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.061463 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" event={"ID":"f4f67336-02d0-46a8-9389-e4a1e2a80ca6","Type":"ContainerStarted","Data":"6ff7f7596e412942ed87aa422daac178ecea6e4ed92a5da2664b679a0947b5f5"} Nov 03 13:04:37.329999 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.329909 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:37.504039 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.503985 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:37.721793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.721758 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:37.882904 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.882873 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:37.886181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:37.886141 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:38.120146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:38.120057 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:38.521804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:38.521772 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:38.718800 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:38.718776 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:39.071048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.070961 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-67jgd" event={"ID":"f4f67336-02d0-46a8-9389-e4a1e2a80ca6","Type":"ContainerStarted","Data":"b19a52126a5f85996a4605cd5f365099c8e12182abe1c79c866d8905ec5e0c64"} Nov 03 13:04:39.072791 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.072767 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6d8j" event={"ID":"99ae820d-6217-425b-9451-247f5dff70dc","Type":"ContainerStarted","Data":"3e1d0eda5f29caeb243b0b8f82d1e5303c1c8ea22d35f789470b43cff712657e"} Nov 03 13:04:39.072930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.072812 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6d8j" event={"ID":"99ae820d-6217-425b-9451-247f5dff70dc","Type":"ContainerStarted","Data":"d2848b0cfc271ab81c831f9b726e66f65355aea20c61ea3c1d19d985f4eea743"} Nov 03 13:04:39.072930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.072838 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:39.074307 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.074283 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6cpnv" event={"ID":"278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2","Type":"ContainerStarted","Data":"10f8fef3e1a5e8d6814d1b4e9ff810f75b60b76023eac4dff737195e2ee5fae6"} Nov 03 13:04:39.074405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.074310 4924 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:04:39.121517 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.121484 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:39.645680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.645644 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:39.760239 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.760210 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:39.902835 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:39.902762 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:40.022374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:40.022342 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:40.324225 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:40.324192 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:40.525625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:40.525586 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:40.635554 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:40.635473 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:40.750125 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:40.750083 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:40.943461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:40.943431 4924 scope.go:122] "RemoveContainer" containerID="ef431b7ea3ee1e21ad2dc173c670342bbc798c5406dcf3f93793556df981e3f1" Nov 03 13:04:41.083282 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:41.083255 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:04:41.083440 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:41.083349 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" event={"ID":"50d1326a-e89d-4e45-90c6-9a254a157e48","Type":"ContainerStarted","Data":"dab9c1855f3b4a6b304858323dce1bc989a0d8d2eaeee67951ce231a711a2f1c"} Nov 03 13:04:41.083631 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:41.083616 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:41.084564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:41.084535 4924 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-2pfz7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" start-of-body= Nov 03 13:04:41.084689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:41.084608 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" podUID="50d1326a-e89d-4e45-90c6-9a254a157e48" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": dial tcp 10.132.0.11:8443: connect: connection refused" Nov 03 13:04:41.125123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:41.125098 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:41.916929 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:41.916897 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:42.090773 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:42.090746 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2pfz7" Nov 03 13:04:42.309891 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:42.309804 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:42.512210 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:42.512171 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:42.541761 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:42.541728 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5564c5bd47-g6p67"] Nov 03 13:04:42.541954 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:42.541935 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" podUID="a8cd572f-aee9-4f80-87c2-2504a0ba404d" Nov 03 13:04:42.648118 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:42.648022 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:42.790160 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:42.790124 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:43.090314 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.090280 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:43.096246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.096225 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:43.125237 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.125205 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:43.139516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.139483 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token\") pod \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " Nov 03 13:04:43.139654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.139534 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration\") pod \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " Nov 03 13:04:43.139654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.139565 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds\") pod \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " Nov 03 13:04:43.139654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.139602 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted\") pod \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " Nov 03 13:04:43.139779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.139654 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates\") pod \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " Nov 03 13:04:43.139779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.139716 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca\") pod \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " Nov 03 13:04:43.139779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.139747 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets\") pod \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " Nov 03 13:04:43.140112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.139975 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted" pod "a8cd572f-aee9-4f80-87c2-2504a0ba404d" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:04:43.140112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.140061 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates" pod "a8cd572f-aee9-4f80-87c2-2504a0ba404d" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:04:43.140266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.140240 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca" pod "a8cd572f-aee9-4f80-87c2-2504a0ba404d" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:04:43.140397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.140374 4924 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-certificates\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:04:43.140623 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.140582 4924 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8cd572f-aee9-4f80-87c2-2504a0ba404d-ca-trust-extracted\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:04:43.142208 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.142184 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration" pod "a8cd572f-aee9-4f80-87c2-2504a0ba404d" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:04:43.142300 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.142237 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds" pod "a8cd572f-aee9-4f80-87c2-2504a0ba404d" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d"). InnerVolumeSpecName "kube-api-access-btfds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:04:43.142300 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.142280 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token" pod "a8cd572f-aee9-4f80-87c2-2504a0ba404d" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:04:43.142400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.142317 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets" pod "a8cd572f-aee9-4f80-87c2-2504a0ba404d" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:04:43.241275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.241245 4924 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-bound-sa-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:04:43.241275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.241273 4924 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-image-registry-private-configuration\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:04:43.241464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.241288 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btfds\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-kube-api-access-btfds\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:04:43.241464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.241298 4924 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8cd572f-aee9-4f80-87c2-2504a0ba404d-trusted-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:04:43.241464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.241307 4924 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8cd572f-aee9-4f80-87c2-2504a0ba404d-installation-pull-secrets\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:04:43.285956 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.285920 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:43.432884 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.432849 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:43.545394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.545361 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:43.719392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.719313 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:43.919775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.919745 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:43.947586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.947556 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:43.947722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.947600 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:43.947722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.947634 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:43.947722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.947656 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:43.947847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.947720 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:43.950193 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.950150 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8425f3a6-d496-41eb-bb83-c65f6011890b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq7lf\" (UID: \"8425f3a6-d496-41eb-bb83-c65f6011890b\") " pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:43.950294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.950215 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"image-registry-896bc546-4fp47\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:43.950294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.950268 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"image-registry-5564c5bd47-g6p67\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:43.950532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.950512 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/05516dd7-e72f-4b64-9707-e401c7e6198b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r2ftp\" (UID: \"05516dd7-e72f-4b64-9707-e401c7e6198b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:43.950570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.950512 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a-metrics-certs\") pod \"network-metrics-daemon-ctkjc\" (UID: \"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a\") " pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:43.980311 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.980241 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" Nov 03 13:04:43.985862 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.985843 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctkjc" Nov 03 13:04:43.989665 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.989644 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:43.992484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:43.992461 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq7lf" Nov 03 13:04:44.048614 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.048439 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") pod \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\" (UID: \"a8cd572f-aee9-4f80-87c2-2504a0ba404d\") " Nov 03 13:04:44.051359 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.051306 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls" pod "a8cd572f-aee9-4f80-87c2-2504a0ba404d" (UID: "a8cd572f-aee9-4f80-87c2-2504a0ba404d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:04:44.094027 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.093998 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5564c5bd47-g6p67" Nov 03 13:04:44.121927 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.121901 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:44.149446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.149418 4924 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8cd572f-aee9-4f80-87c2-2504a0ba404d-registry-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:04:44.257419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.257337 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6wbc" Nov 03 13:04:44.306685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.306651 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5564c5bd47-g6p67"] Nov 03 13:04:44.311792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.311762 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5564c5bd47-g6p67"] Nov 03 13:04:44.355794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.355766 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ctkjc"] Nov 03 13:04:44.358820 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:44.358793 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9d3b1a_170f_4a5d_9a6f_2460ce85ca7a.slice/crio-d4653962ee4bb5e30d911ae0db9e6cac55478704239ad42154c196ddd67e154f WatchSource:0}: Error finding container d4653962ee4bb5e30d911ae0db9e6cac55478704239ad42154c196ddd67e154f: Status 404 returned error can't find the container with id d4653962ee4bb5e30d911ae0db9e6cac55478704239ad42154c196ddd67e154f Nov 03 13:04:44.385306 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.385281 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp"] Nov 03 13:04:44.408667 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.408610 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-896bc546-4fp47"] Nov 03 13:04:44.410541 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.410516 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cq7lf"] Nov 03 13:04:44.411688 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:44.411660 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56168f93_9536_4108_a5ce_5677c95ecf0c.slice/crio-7c1b37a8ee95f829337804b72904015ca4e531d3421a951962735ccdc4ef291c WatchSource:0}: Error finding container 7c1b37a8ee95f829337804b72904015ca4e531d3421a951962735ccdc4ef291c: Status 404 returned error can't find the container with id 7c1b37a8ee95f829337804b72904015ca4e531d3421a951962735ccdc4ef291c Nov 03 13:04:44.414759 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:44.414736 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8425f3a6_d496_41eb_bb83_c65f6011890b.slice/crio-7888cbc580b6e005d05f44bd05db1b33d728009e7723c45ced1b94d292793d86 WatchSource:0}: Error finding container 7888cbc580b6e005d05f44bd05db1b33d728009e7723c45ced1b94d292793d86: Status 404 returned error can't find the container with id 7888cbc580b6e005d05f44bd05db1b33d728009e7723c45ced1b94d292793d86 Nov 03 13:04:44.785530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.785495 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:44.785692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.785665 4924 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:04:44.786803 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.786778 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8478467d49-l5dtm" Nov 03 13:04:44.947662 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:44.947628 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cd572f-aee9-4f80-87c2-2504a0ba404d" path="/var/lib/kubelet/pods/a8cd572f-aee9-4f80-87c2-2504a0ba404d/volumes" Nov 03 13:04:45.099010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:45.098924 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" event={"ID":"05516dd7-e72f-4b64-9707-e401c7e6198b","Type":"ContainerStarted","Data":"e6caa71700be8b83ed1e767dc1aa2c0c4f7908ba037d641bddbc573ee401e828"} Nov 03 13:04:45.100533 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:45.100504 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq7lf" event={"ID":"8425f3a6-d496-41eb-bb83-c65f6011890b","Type":"ContainerStarted","Data":"e8e0619b2f9c666da47421117ecfb632df9d73eaa38524698133b4bb46de753c"} Nov 03 13:04:45.100668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:45.100572 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq7lf" event={"ID":"8425f3a6-d496-41eb-bb83-c65f6011890b","Type":"ContainerStarted","Data":"7888cbc580b6e005d05f44bd05db1b33d728009e7723c45ced1b94d292793d86"} Nov 03 13:04:45.102227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:45.102202 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-896bc546-4fp47" event={"ID":"56168f93-9536-4108-a5ce-5677c95ecf0c","Type":"ContainerStarted","Data":"6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13"} Nov 03 13:04:45.102321 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:45.102244 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-896bc546-4fp47" event={"ID":"56168f93-9536-4108-a5ce-5677c95ecf0c","Type":"ContainerStarted","Data":"7c1b37a8ee95f829337804b72904015ca4e531d3421a951962735ccdc4ef291c"} Nov 03 13:04:45.102408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:45.102377 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:04:45.103808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:45.103787 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctkjc" event={"ID":"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a","Type":"ContainerStarted","Data":"d4653962ee4bb5e30d911ae0db9e6cac55478704239ad42154c196ddd67e154f"} Nov 03 13:04:46.110561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:46.110528 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq7lf" event={"ID":"8425f3a6-d496-41eb-bb83-c65f6011890b","Type":"ContainerStarted","Data":"c795a3f758583cf5e1c43750f8308c862cba3bd64db5eeb753fdf04c40d70ce4"} Nov 03 13:04:47.116891 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:47.116802 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctkjc" event={"ID":"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a","Type":"ContainerStarted","Data":"68fe156453f6426fe968dfb26842a1cfd3c480be083d7b4c320296ed2044b18f"} Nov 03 13:04:47.116891 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:47.116886 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctkjc" event={"ID":"cf9d3b1a-170f-4a5d-9a6f-2460ce85ca7a","Type":"ContainerStarted","Data":"ce7bfbab1df075823bb95697f30122cb65f8f65551c0881069d37bd2139c71a6"} Nov 03 13:04:47.118968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:47.118938 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" event={"ID":"05516dd7-e72f-4b64-9707-e401c7e6198b","Type":"ContainerStarted","Data":"577f93e3b5c8f2a8cce54fa276b70542e29d360c8aec256d5d7c8dc4044c067b"} Nov 03 13:04:47.119098 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:47.118992 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r2ftp" event={"ID":"05516dd7-e72f-4b64-9707-e401c7e6198b","Type":"ContainerStarted","Data":"08a3647c07a9421e3004a4d7b738aa648a3cee0816a828b90fcce8c455cb7f11"} Nov 03 13:04:47.890205 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:47.890171 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:48.004046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:48.003963 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:48.117704 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:48.117667 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:48.124486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:48.124451 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq7lf" event={"ID":"8425f3a6-d496-41eb-bb83-c65f6011890b","Type":"ContainerStarted","Data":"d636ab4897837ad32f3e00f67ba5ca0ae655c12c046d1a0dbbeaee00383f83fb"} Nov 03 13:04:48.276899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:48.276815 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:48.560495 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:04:48.560418 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:04:49.080637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:49.080608 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k6d8j" Nov 03 13:04:51.319723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:51.319686 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:51.430574 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:51.430536 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:53.137622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.137582 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:53.424051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.424016 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:53.536374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.536333 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:53.658448 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.658407 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:53.770911 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.770839 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:53.811935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.811906 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nh6c9" Nov 03 13:04:53.815080 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.815047 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vhgsp"] Nov 03 13:04:53.847414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.847389 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.929736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.929706 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-tls\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.929908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.929757 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-wtmp\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.929908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.929778 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.930018 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.929906 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74fwq\" (UniqueName: \"kubernetes.io/projected/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-kube-api-access-74fwq\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.930018 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.929936 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.930018 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.929957 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-root\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.930018 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.930008 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-textfile\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.930246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.930051 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-metrics-client-ca\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.930246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.930183 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-sys\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:53.969385 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.969355 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 13:04:53.969550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.969416 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5rlch\"" Nov 03 13:04:53.969550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.969425 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 13:04:53.983536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:53.983507 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:54.012750 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.012725 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 13:04:54.012953 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.012937 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 13:04:54.031207 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031144 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-sys\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031207 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031177 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-tls\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031208 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-wtmp\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031235 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031266 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-sys\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031297 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74fwq\" (UniqueName: \"kubernetes.io/projected/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-kube-api-access-74fwq\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031325 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031339 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-wtmp\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031355 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-root\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031378 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-textfile\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031403 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-metrics-client-ca\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031433 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-root\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031859 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-textfile\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.031970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031946 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-metrics-client-ca\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.032023 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.031974 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-accelerators-collector-config\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.033557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.033529 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.033718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.033699 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-node-exporter-tls\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.084697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.084666 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74fwq\" (UniqueName: \"kubernetes.io/projected/aee646c2-b8d9-4998-b6e4-cffbfba7ea94-kube-api-access-74fwq\") pod \"node-exporter-vhgsp\" (UID: \"aee646c2-b8d9-4998-b6e4-cffbfba7ea94\") " pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.196302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.196263 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vhgsp" Nov 03 13:04:54.218268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.218235 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:54.229408 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:04:54.229385 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee646c2_b8d9_4998_b6e4_cffbfba7ea94.slice/crio-5af00b4da5ab49a6928fdc8677b2ed12a7521fb0a5b8c21602fbe9481d46eb9f WatchSource:0}: Error finding container 5af00b4da5ab49a6928fdc8677b2ed12a7521fb0a5b8c21602fbe9481d46eb9f: Status 404 returned error can't find the container with id 5af00b4da5ab49a6928fdc8677b2ed12a7521fb0a5b8c21602fbe9481d46eb9f Nov 03 13:04:54.596590 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:54.596556 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:55.063733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:55.063700 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:55.153773 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:55.153736 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhgsp" event={"ID":"aee646c2-b8d9-4998-b6e4-cffbfba7ea94","Type":"ContainerStarted","Data":"5af00b4da5ab49a6928fdc8677b2ed12a7521fb0a5b8c21602fbe9481d46eb9f"} Nov 03 13:04:55.230276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:55.230238 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:55.928789 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:55.928754 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:56.043766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:56.043737 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:57.162031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:57.161996 4924 generic.go:356] "Generic (PLEG): container finished" podID="aee646c2-b8d9-4998-b6e4-cffbfba7ea94" containerID="82f27c353ff83a101c00a5f7dcf33fda3d782a06e3448e3db373d44c4bc9ac48" exitCode=0 Nov 03 13:04:57.162427 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:57.162086 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhgsp" event={"ID":"aee646c2-b8d9-4998-b6e4-cffbfba7ea94","Type":"ContainerDied","Data":"82f27c353ff83a101c00a5f7dcf33fda3d782a06e3448e3db373d44c4bc9ac48"} Nov 03 13:04:57.586623 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:57.586548 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:57.699740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:57.699703 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:57.812611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:57.812583 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:57.928770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:57.928737 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:58.087812 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:58.087777 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:58.168650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:58.168611 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhgsp" event={"ID":"aee646c2-b8d9-4998-b6e4-cffbfba7ea94","Type":"ContainerStarted","Data":"3696617872cedcb3eabf1ec314a42681decda6c757ac185d093d024a053ec452"} Nov 03 13:04:58.169100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:58.168716 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vhgsp" event={"ID":"aee646c2-b8d9-4998-b6e4-cffbfba7ea94","Type":"ContainerStarted","Data":"b20a9c1d0a7bc1f6d841945b39f568b70faa04de869379181373a6d60bf16352"} Nov 03 13:04:58.208857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:58.208795 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:58.354718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:58.354678 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:58.471769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:58.471697 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:58.816391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:58.816325 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:59.447355 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:59.447318 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:59.563141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:04:59.563106 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:02.821771 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:02.821731 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:04.177489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:04.177449 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:04.289474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:04.289443 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:04.841709 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:04.841662 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:05.204720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:05.204658 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vhgsp" podStartSLOduration=10.21615934 podStartE2EDuration="12.204644073s" totalImagesPullingTime="1.921168085s" totalInitContainerRuntime="67.316648ms" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:58.29516972 +0000 UTC m=+31.489573324" watchObservedRunningTime="2026-11-03 13:05:05.204644073 +0000 UTC m=+38.399047701" Nov 03 13:05:05.205258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:05.205235 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-896bc546-4fp47"] Nov 03 13:05:05.209633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:05.209613 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:05:05.239433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:05.239407 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:05.349635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:05.349603 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:05.986394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:05.986357 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:06.103802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:06.103769 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:06.297759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:06.297685 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:06.410220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:06.410190 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:10.395133 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:10.395043 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:10.796152 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:10.796122 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:10.908817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:10.908781 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:11.021063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:11.020992 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:14.141418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:14.141380 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:14.265690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:14.265637 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:15.284872 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:15.284838 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:15.397718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:15.397687 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:16.872707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:16.872670 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:17.016094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:17.016042 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:18.557374 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:05:18.557334 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:05:21.307661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:21.307624 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:21.431129 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:21.431091 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:24.250846 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:24.250811 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:28.130410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:28.130372 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:28.246319 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:28.246286 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:30.224709 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.224648 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-896bc546-4fp47" podUID="56168f93-9536-4108-a5ce-5677c95ecf0c" containerName="registry" containerID="cri-o://6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13" gracePeriod=30 Nov 03 13:05:30.474594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.474572 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:05:30.515406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515344 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn\") pod \"56168f93-9536-4108-a5ce-5677c95ecf0c\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " Nov 03 13:05:30.515406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515387 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token\") pod \"56168f93-9536-4108-a5ce-5677c95ecf0c\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " Nov 03 13:05:30.515604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515406 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") pod \"56168f93-9536-4108-a5ce-5677c95ecf0c\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " Nov 03 13:05:30.515604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515438 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted\") pod \"56168f93-9536-4108-a5ce-5677c95ecf0c\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " Nov 03 13:05:30.515604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515491 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca\") pod \"56168f93-9536-4108-a5ce-5677c95ecf0c\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " Nov 03 13:05:30.515604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515542 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets\") pod \"56168f93-9536-4108-a5ce-5677c95ecf0c\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " Nov 03 13:05:30.516181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515605 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates\") pod \"56168f93-9536-4108-a5ce-5677c95ecf0c\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " Nov 03 13:05:30.516181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515645 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration\") pod \"56168f93-9536-4108-a5ce-5677c95ecf0c\" (UID: \"56168f93-9536-4108-a5ce-5677c95ecf0c\") " Nov 03 13:05:30.516181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.515982 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca" pod "56168f93-9536-4108-a5ce-5677c95ecf0c" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:30.516465 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.516438 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates" pod "56168f93-9536-4108-a5ce-5677c95ecf0c" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:30.517979 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.517954 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn" pod "56168f93-9536-4108-a5ce-5677c95ecf0c" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c"). InnerVolumeSpecName "kube-api-access-znzwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:30.518126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.518091 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls" pod "56168f93-9536-4108-a5ce-5677c95ecf0c" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:30.518234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.518122 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token" pod "56168f93-9536-4108-a5ce-5677c95ecf0c" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:30.518234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.518172 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration" pod "56168f93-9536-4108-a5ce-5677c95ecf0c" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:30.518370 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.518346 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets" pod "56168f93-9536-4108-a5ce-5677c95ecf0c" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:30.522626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.522602 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted" pod "56168f93-9536-4108-a5ce-5677c95ecf0c" (UID: "56168f93-9536-4108-a5ce-5677c95ecf0c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:05:30.616329 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.616294 4924 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-certificates\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:30.616329 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.616323 4924 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-image-registry-private-configuration\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:30.616329 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.616333 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzwn\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-kube-api-access-znzwn\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:30.616548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.616345 4924 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-bound-sa-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:30.616548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.616353 4924 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56168f93-9536-4108-a5ce-5677c95ecf0c-registry-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:30.616548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.616362 4924 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56168f93-9536-4108-a5ce-5677c95ecf0c-ca-trust-extracted\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:30.616548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.616371 4924 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56168f93-9536-4108-a5ce-5677c95ecf0c-trusted-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:30.616548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:30.616380 4924 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56168f93-9536-4108-a5ce-5677c95ecf0c-installation-pull-secrets\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:31.288957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.288867 4924 generic.go:356] "Generic (PLEG): container finished" podID="56168f93-9536-4108-a5ce-5677c95ecf0c" containerID="6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13" exitCode=0 Nov 03 13:05:31.288957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.288916 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-896bc546-4fp47" event={"ID":"56168f93-9536-4108-a5ce-5677c95ecf0c","Type":"ContainerDied","Data":"6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13"} Nov 03 13:05:31.288957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.288917 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896bc546-4fp47" Nov 03 13:05:31.288957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.288957 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-896bc546-4fp47" event={"ID":"56168f93-9536-4108-a5ce-5677c95ecf0c","Type":"ContainerDied","Data":"7c1b37a8ee95f829337804b72904015ca4e531d3421a951962735ccdc4ef291c"} Nov 03 13:05:31.289570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.288979 4924 scope.go:122] "RemoveContainer" containerID="6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13" Nov 03 13:05:31.298136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.298119 4924 scope.go:122] "RemoveContainer" containerID="6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13" Nov 03 13:05:31.298419 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:05:31.298395 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13\": container with ID starting with 6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13 not found: ID does not exist" containerID="6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13" Nov 03 13:05:31.298490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.298432 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13"} err="failed to get container status \"6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13\": rpc error: code = NotFound desc = could not find container \"6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13\": container with ID starting with 6a00ee344aa361bee3c31b85b8678798c356bd22ed716a6c0f59335fcc2b5b13 not found: ID does not exist" Nov 03 13:05:31.489960 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.489922 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-896bc546-4fp47"] Nov 03 13:05:31.536425 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:31.536392 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-896bc546-4fp47"] Nov 03 13:05:32.948108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:32.948051 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56168f93-9536-4108-a5ce-5677c95ecf0c" path="/var/lib/kubelet/pods/56168f93-9536-4108-a5ce-5677c95ecf0c/volumes" Nov 03 13:05:42.333997 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:42.333961 4924 generic.go:356] "Generic (PLEG): container finished" podID="71f66428-6b35-45cc-ac46-ce5c07aa958c" containerID="6d1ceb0463223c01ba86329bf485d3fcc60b314381623d4f3fe5dc28020ea909" exitCode=0 Nov 03 13:05:42.334970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:42.334036 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" event={"ID":"71f66428-6b35-45cc-ac46-ce5c07aa958c","Type":"ContainerDied","Data":"6d1ceb0463223c01ba86329bf485d3fcc60b314381623d4f3fe5dc28020ea909"} Nov 03 13:05:42.334970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:42.334427 4924 scope.go:122] "RemoveContainer" containerID="6d1ceb0463223c01ba86329bf485d3fcc60b314381623d4f3fe5dc28020ea909" Nov 03 13:05:43.340038 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:43.339999 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-zphsk" event={"ID":"71f66428-6b35-45cc-ac46-ce5c07aa958c","Type":"ContainerStarted","Data":"c1a607c773a9dbb82b930042f5d499cfa34f731a760ab4b98e54669e74bb5572"} Nov 03 13:05:44.211646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:44.211588 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:44.447503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:44.447466 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:48.557957 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:05:48.557889 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:05:51.301854 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:51.301802 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:51.415787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:51.415752 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:54.110921 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:54.110865 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:54.219967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:54.219930 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:54.469930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:54.469897 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:54.584712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:54.584680 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:56.178512 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:56.178475 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:56.287968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:56.287930 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:05:58.392817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:58.392781 4924 generic.go:356] "Generic (PLEG): container finished" podID="f70a3801-ffa0-4049-aeee-0d886ccb30b3" containerID="27781848cbbbbfa98733d1cb8339cb56c64b7c45ff8a23a0958d93c05e56d05e" exitCode=0 Nov 03 13:05:58.393250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:58.392854 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" event={"ID":"f70a3801-ffa0-4049-aeee-0d886ccb30b3","Type":"ContainerDied","Data":"27781848cbbbbfa98733d1cb8339cb56c64b7c45ff8a23a0958d93c05e56d05e"} Nov 03 13:05:58.393320 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:58.393294 4924 scope.go:122] "RemoveContainer" containerID="27781848cbbbbfa98733d1cb8339cb56c64b7c45ff8a23a0958d93c05e56d05e" Nov 03 13:05:58.394483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:58.394459 4924 generic.go:356] "Generic (PLEG): container finished" podID="52b2bd33-97be-42ee-b7d4-2b86f1de3d7d" containerID="8a2449557043cfc2f0b4374edd44964b1e13ac7f57a9e24cb784c08f47791bfd" exitCode=0 Nov 03 13:05:58.394587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:58.394490 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xhtjf" event={"ID":"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d","Type":"ContainerDied","Data":"8a2449557043cfc2f0b4374edd44964b1e13ac7f57a9e24cb784c08f47791bfd"} Nov 03 13:05:58.394770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:58.394757 4924 scope.go:122] "RemoveContainer" containerID="8a2449557043cfc2f0b4374edd44964b1e13ac7f57a9e24cb784c08f47791bfd" Nov 03 13:05:59.400645 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:59.400609 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg68w" event={"ID":"f70a3801-ffa0-4049-aeee-0d886ccb30b3","Type":"ContainerStarted","Data":"aa3e1f3ffc491991adfdfeddfbb61fdcce8422280660647cf3a35336328f27a7"} Nov 03 13:05:59.403859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:05:59.403835 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xhtjf" event={"ID":"52b2bd33-97be-42ee-b7d4-2b86f1de3d7d","Type":"ContainerStarted","Data":"4751fd541c5c151b92229931515377e28aa958ad6057eda95a88758e1437c836"} Nov 03 13:06:11.878781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:11.878743 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:11.989731 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:11.989700 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:12.434857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:12.434822 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:12.545473 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:12.545437 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:12.793966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:12.793881 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:12.905087 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:12.905034 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:17.193216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:17.193178 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:17.441324 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:17.441289 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:18.558102 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:06:18.558038 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:06:24.629380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:24.629343 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:24.739217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:24.739177 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:36.319517 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:36.319475 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:36.431712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:06:36.431676 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:06:48.559296 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:06:48.559212 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:07:05.512850 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:07:05.512813 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:07:17.515473 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:07:17.515434 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:07:18.556909 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:07:18.556869 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:07:48.556539 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:07:48.556500 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:07:59.562931 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:07:59.562892 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:07:59.676018 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:07:59.675988 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:17.845191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:17.845105 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:18.556769 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:08:18.556731 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:08:45.407877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:45.407846 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:45.661345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:45.661261 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:46.706310 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.706277 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl"] Nov 03 13:08:46.709925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.707445 4924 cpu_manager.go:414] "containerMap: removing container" podUID="56168f93-9536-4108-a5ce-5677c95ecf0c" containerName="registry" Nov 03 13:08:46.709925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.707490 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56168f93-9536-4108-a5ce-5677c95ecf0c" containerName="registry" Nov 03 13:08:46.709925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.707675 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="56168f93-9536-4108-a5ce-5677c95ecf0c" containerName="registry" Nov 03 13:08:46.712145 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.712117 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.768763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.768724 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.768911 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.768778 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg96s\" (UniqueName: \"kubernetes.io/projected/f8c29c94-6386-4edb-a833-07f7e3d824d1-kube-api-access-fg96s\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.768911 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.768854 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.809699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.809658 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 13:08:46.809849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.809789 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 13:08:46.810895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.810877 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl"] Nov 03 13:08:46.811721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.811693 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd"] Nov 03 13:08:46.814909 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.814894 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.856599 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.856573 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ldmbb\"" Nov 03 13:08:46.864344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.864321 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jpxs9\"" Nov 03 13:08:46.869298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.869271 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.869430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.869310 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.869430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.869341 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.869430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.869373 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg96s\" (UniqueName: \"kubernetes.io/projected/f8c29c94-6386-4edb-a833-07f7e3d824d1-kube-api-access-fg96s\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.869430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.869415 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxf5\" (UniqueName: \"kubernetes.io/projected/d6690ccb-c514-4b2b-9d2d-8a580995f308-kube-api-access-5zxf5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.869645 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.869548 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.869807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.869789 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.869898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.869870 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.870025 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.870010 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd"] Nov 03 13:08:46.924479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.924452 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg96s\" (UniqueName: \"kubernetes.io/projected/f8c29c94-6386-4edb-a833-07f7e3d824d1-kube-api-access-fg96s\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:46.970684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.970596 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.970684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.970649 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.970925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.970686 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxf5\" (UniqueName: \"kubernetes.io/projected/d6690ccb-c514-4b2b-9d2d-8a580995f308-kube-api-access-5zxf5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.971033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.971012 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.971110 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.971090 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:46.988897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:46.988870 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:47.025530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.025500 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxf5\" (UniqueName: \"kubernetes.io/projected/d6690ccb-c514-4b2b-9d2d-8a580995f308-kube-api-access-5zxf5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:47.026758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.026742 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:08:47.130754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.130722 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:08:47.391708 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.391632 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl"] Nov 03 13:08:47.394889 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:08:47.394863 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c29c94_6386_4edb_a833_07f7e3d824d1.slice/crio-6b79ea5497bae6e27b7c82860789db2ebb4f094c230eab1fb71a913ea7ddec4e WatchSource:0}: Error finding container 6b79ea5497bae6e27b7c82860789db2ebb4f094c230eab1fb71a913ea7ddec4e: Status 404 returned error can't find the container with id 6b79ea5497bae6e27b7c82860789db2ebb4f094c230eab1fb71a913ea7ddec4e Nov 03 13:08:47.489389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.489359 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd"] Nov 03 13:08:47.492703 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:08:47.492677 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6690ccb_c514_4b2b_9d2d_8a580995f308.slice/crio-96abac70b9487302fce6ab9090bd93934f080951ab5791dd2edd2e674b958884 WatchSource:0}: Error finding container 96abac70b9487302fce6ab9090bd93934f080951ab5791dd2edd2e674b958884: Status 404 returned error can't find the container with id 96abac70b9487302fce6ab9090bd93934f080951ab5791dd2edd2e674b958884 Nov 03 13:08:47.751739 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.751685 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:47.993939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.993906 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" event={"ID":"f8c29c94-6386-4edb-a833-07f7e3d824d1","Type":"ContainerStarted","Data":"6b79ea5497bae6e27b7c82860789db2ebb4f094c230eab1fb71a913ea7ddec4e"} Nov 03 13:08:47.994924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.994896 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:47.995177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:47.995157 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" event={"ID":"d6690ccb-c514-4b2b-9d2d-8a580995f308","Type":"ContainerStarted","Data":"96abac70b9487302fce6ab9090bd93934f080951ab5791dd2edd2e674b958884"} Nov 03 13:08:48.565730 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:08:48.565687 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:08:49.008795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.008758 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx"] Nov 03 13:08:49.012464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.012430 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.065715 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.065685 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx"] Nov 03 13:08:49.091191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.091157 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.091380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.091234 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.091458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.091405 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwwc\" (UniqueName: \"kubernetes.io/projected/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-kube-api-access-vnwwc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.110985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.110950 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-tw7pn\"" Nov 03 13:08:49.191916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.191877 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwwc\" (UniqueName: \"kubernetes.io/projected/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-kube-api-access-vnwwc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.192165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.191929 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.192165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.191986 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.192501 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.192477 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.192608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.192584 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.247647 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.247606 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwwc\" (UniqueName: \"kubernetes.io/projected/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-kube-api-access-vnwwc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.332437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.332348 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:08:49.712681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:49.711656 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx"] Nov 03 13:08:49.714398 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:08:49.714368 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e0ec10_ffe1_426d_a0a2_0c902d35c6da.slice/crio-8429a6d0e079a80af84837002d749a3727058c6b4f3fd6bc06acfb3527fb9d3f WatchSource:0}: Error finding container 8429a6d0e079a80af84837002d749a3727058c6b4f3fd6bc06acfb3527fb9d3f: Status 404 returned error can't find the container with id 8429a6d0e079a80af84837002d749a3727058c6b4f3fd6bc06acfb3527fb9d3f Nov 03 13:08:50.006924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.006836 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" event={"ID":"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da","Type":"ContainerStarted","Data":"8429a6d0e079a80af84837002d749a3727058c6b4f3fd6bc06acfb3527fb9d3f"} Nov 03 13:08:50.469244 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.469207 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6"] Nov 03 13:08:50.474440 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.474414 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.502230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.502200 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.502379 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.502239 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.502379 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.502294 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjkg\" (UniqueName: \"kubernetes.io/projected/288531c0-4667-47fe-b92a-f556d59a0594-kube-api-access-dkjkg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.554799 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.554767 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-kfswc\"" Nov 03 13:08:50.556391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.556362 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6"] Nov 03 13:08:50.583133 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.583096 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt"] Nov 03 13:08:50.587163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.587129 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.603706 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.603679 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.603841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.603716 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.603841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.603778 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjkg\" (UniqueName: \"kubernetes.io/projected/288531c0-4667-47fe-b92a-f556d59a0594-kube-api-access-dkjkg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.604182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.604154 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.604813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.604791 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.639139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.639106 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-9btg6\"" Nov 03 13:08:50.643546 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.643507 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt"] Nov 03 13:08:50.659126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.658969 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjkg\" (UniqueName: \"kubernetes.io/projected/288531c0-4667-47fe-b92a-f556d59a0594-kube-api-access-dkjkg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.662835 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.662806 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg"] Nov 03 13:08:50.666470 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.666448 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.704587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.704555 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.704735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.704606 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.704735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.704704 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5k7r\" (UniqueName: \"kubernetes.io/projected/c66682e7-3106-4c0f-8a27-caec24edaef2-kube-api-access-z5k7r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.704846 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.704753 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7m9z\" (UniqueName: \"kubernetes.io/projected/2713157b-b9a5-4cff-a3ba-0696ed444ab9-kube-api-access-p7m9z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.704846 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.704797 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.704846 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.704826 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.717392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.717368 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-pnzd5\"" Nov 03 13:08:50.724348 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.724283 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg"] Nov 03 13:08:50.765687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.765651 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl"] Nov 03 13:08:50.769860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.769837 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.791630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.791606 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:08:50.806159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806124 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.806258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806179 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.806258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806222 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.806365 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806256 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.806365 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806294 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.806365 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806347 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.806478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806402 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tm87\" (UniqueName: \"kubernetes.io/projected/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-kube-api-access-7tm87\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.806478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806437 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5k7r\" (UniqueName: \"kubernetes.io/projected/c66682e7-3106-4c0f-8a27-caec24edaef2-kube-api-access-z5k7r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.806478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.806463 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7m9z\" (UniqueName: \"kubernetes.io/projected/2713157b-b9a5-4cff-a3ba-0696ed444ab9-kube-api-access-p7m9z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.807376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.807242 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.807600 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.807566 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.807811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.807782 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.808036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.808017 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.819128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.819106 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-pw5j2\"" Nov 03 13:08:50.824876 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.824580 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl"] Nov 03 13:08:50.865213 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.864674 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7m9z\" (UniqueName: \"kubernetes.io/projected/2713157b-b9a5-4cff-a3ba-0696ed444ab9-kube-api-access-p7m9z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:50.865213 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.865175 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5k7r\" (UniqueName: \"kubernetes.io/projected/c66682e7-3106-4c0f-8a27-caec24edaef2-kube-api-access-z5k7r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.907597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.907570 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:08:50.907802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.907773 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.907846 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.907833 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.907914 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.907897 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tm87\" (UniqueName: \"kubernetes.io/projected/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-kube-api-access-7tm87\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.908248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.908223 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.908560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.908536 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.965945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.964306 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tm87\" (UniqueName: \"kubernetes.io/projected/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-kube-api-access-7tm87\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:50.986402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:50.986374 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:08:51.094316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:51.094253 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:08:51.167023 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:51.166902 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6"] Nov 03 13:08:51.289034 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:51.288962 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt"] Nov 03 13:08:51.293293 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:08:51.293261 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66682e7_3106_4c0f_8a27_caec24edaef2.slice/crio-e2e517ac90007a83b3b9574cc25d68200f7d93f9822a103f4b80c5c272d1f778 WatchSource:0}: Error finding container e2e517ac90007a83b3b9574cc25d68200f7d93f9822a103f4b80c5c272d1f778: Status 404 returned error can't find the container with id e2e517ac90007a83b3b9574cc25d68200f7d93f9822a103f4b80c5c272d1f778 Nov 03 13:08:51.373596 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:51.373564 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg"] Nov 03 13:08:51.377302 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:08:51.377270 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2713157b_b9a5_4cff_a3ba_0696ed444ab9.slice/crio-5586bd70cd12196c3e9d372a63b1f347cc6fc1ce1db7dc2463f2ea99a5198ca8 WatchSource:0}: Error finding container 5586bd70cd12196c3e9d372a63b1f347cc6fc1ce1db7dc2463f2ea99a5198ca8: Status 404 returned error can't find the container with id 5586bd70cd12196c3e9d372a63b1f347cc6fc1ce1db7dc2463f2ea99a5198ca8 Nov 03 13:08:51.478222 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:51.478179 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl"] Nov 03 13:08:51.482324 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:08:51.482296 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8ccbfa_d9a5_4989_b3b2_1992bd5f6249.slice/crio-9afb1779f34350e67c0bb90204027cdbe9cfdf77997575c00ff13e9f03c04551 WatchSource:0}: Error finding container 9afb1779f34350e67c0bb90204027cdbe9cfdf77997575c00ff13e9f03c04551: Status 404 returned error can't find the container with id 9afb1779f34350e67c0bb90204027cdbe9cfdf77997575c00ff13e9f03c04551 Nov 03 13:08:52.019170 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.019133 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" event={"ID":"c66682e7-3106-4c0f-8a27-caec24edaef2","Type":"ContainerStarted","Data":"e2e517ac90007a83b3b9574cc25d68200f7d93f9822a103f4b80c5c272d1f778"} Nov 03 13:08:52.020894 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.020840 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" event={"ID":"288531c0-4667-47fe-b92a-f556d59a0594","Type":"ContainerStarted","Data":"38dddd4100b7b3c93d2c96ee84c65460d2c721f0a951abc4254b64b49540f30f"} Nov 03 13:08:52.023712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.023643 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" event={"ID":"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249","Type":"ContainerStarted","Data":"9afb1779f34350e67c0bb90204027cdbe9cfdf77997575c00ff13e9f03c04551"} Nov 03 13:08:52.025221 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.025176 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" event={"ID":"2713157b-b9a5-4cff-a3ba-0696ed444ab9","Type":"ContainerStarted","Data":"5586bd70cd12196c3e9d372a63b1f347cc6fc1ce1db7dc2463f2ea99a5198ca8"} Nov 03 13:08:52.448973 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.448916 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk"] Nov 03 13:08:52.454347 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.454319 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.503777 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.503706 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk"] Nov 03 13:08:52.524763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.524490 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.524763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.524561 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.524763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.524631 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5q79\" (UniqueName: \"kubernetes.io/projected/aba43d56-800d-44a3-9092-68d1a6fe20cb-kube-api-access-p5q79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.626916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.625709 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5q79\" (UniqueName: \"kubernetes.io/projected/aba43d56-800d-44a3-9092-68d1a6fe20cb-kube-api-access-p5q79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.626916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.625797 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.626916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.625840 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.626916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.626273 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.626916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.626862 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.697320 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.697285 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5q79\" (UniqueName: \"kubernetes.io/projected/aba43d56-800d-44a3-9092-68d1a6fe20cb-kube-api-access-p5q79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.760814 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.759238 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586"] Nov 03 13:08:52.774302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.774274 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.776065 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.775860 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:08:52.823050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.822989 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586"] Nov 03 13:08:52.829910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.827663 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjb8x\" (UniqueName: \"kubernetes.io/projected/3d512678-9504-4ce7-8479-ba09349077e8-kube-api-access-tjb8x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.829910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.827712 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.829910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.827743 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.928270 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.928228 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.928427 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.928310 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.928427 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.928408 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjb8x\" (UniqueName: \"kubernetes.io/projected/3d512678-9504-4ce7-8479-ba09349077e8-kube-api-access-tjb8x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.929192 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.929164 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.929471 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.929447 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:52.993629 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:52.993593 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjb8x\" (UniqueName: \"kubernetes.io/projected/3d512678-9504-4ce7-8479-ba09349077e8-kube-api-access-tjb8x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:53.094354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.094278 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:53.114608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.114575 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:08:53.201966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.201886 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk"] Nov 03 13:08:53.419065 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.418987 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:53.456549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.454778 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8"] Nov 03 13:08:53.460639 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.459966 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.516866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.516750 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 13:08:53.516866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.516840 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wd2hr\"" Nov 03 13:08:53.524718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.524620 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8"] Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538387 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538453 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538497 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgfp\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-kube-api-access-lkgfp\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538541 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538567 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538591 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538628 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538686 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.538906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.538711 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.541703 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.541586 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586"] Nov 03 13:08:53.631015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.630946 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.639799 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.639845 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.639931 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.639972 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.640041 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.640065 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.640203 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.640265 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.640316 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgfp\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-kube-api-access-lkgfp\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.641142 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.641706 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.641982 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.643437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.643100 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.644557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.644502 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.646195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.646165 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.648014 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.647991 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.702668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.702553 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.707323 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.707274 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgfp\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-kube-api-access-lkgfp\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-shvr8\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.779686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.777606 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95"] Nov 03 13:08:53.780093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.780042 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:08:53.783541 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.783513 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.838252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.838204 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-755z8\"" Nov 03 13:08:53.839049 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.839022 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95"] Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842156 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842220 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842250 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bh9\" (UniqueName: \"kubernetes.io/projected/0db0b8cf-03c4-4493-a3f8-464723ef7eec-kube-api-access-67bh9\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842280 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842314 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842339 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842372 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842443 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.842694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.842491 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.850837 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.850809 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:53.943783 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.943698 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.943962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.943804 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.943962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.943837 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.943962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.943877 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.943962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.943895 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bh9\" (UniqueName: \"kubernetes.io/projected/0db0b8cf-03c4-4493-a3f8-464723ef7eec-kube-api-access-67bh9\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.943962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.943925 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.944307 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.943962 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.944307 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.944086 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.944307 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.944115 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.944703 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.944633 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.945586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.945426 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.946109 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.946088 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.946392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.946373 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.946686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.946598 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.972391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.972287 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:53.973167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.973064 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:08:53.987103 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:53.987050 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:54.003626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:54.003550 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db0b8cf-03c4-4493-a3f8-464723ef7eec-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:54.005809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:54.005763 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bh9\" (UniqueName: \"kubernetes.io/projected/0db0b8cf-03c4-4493-a3f8-464723ef7eec-kube-api-access-67bh9\") pod \"openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95\" (UID: \"0db0b8cf-03c4-4493-a3f8-464723ef7eec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:54.040797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:54.040674 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" event={"ID":"aba43d56-800d-44a3-9092-68d1a6fe20cb","Type":"ContainerStarted","Data":"e050c0dac947c7ce33d5d37a940e2080a566f91ff30c7a2866c4dc9de4f72efd"} Nov 03 13:08:54.043921 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:54.043849 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" event={"ID":"3d512678-9504-4ce7-8479-ba09349077e8","Type":"ContainerStarted","Data":"f430264d414d2bfe159de3f643beecd99670424c69c805197e0659b507831d32"} Nov 03 13:08:54.122750 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:54.122388 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:08:54.223204 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:54.222342 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8"] Nov 03 13:08:54.532465 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:54.532298 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95"] Nov 03 13:08:54.543374 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:08:54.543330 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db0b8cf_03c4_4493_a3f8_464723ef7eec.slice/crio-8dc3f58a51628a1ba1e51efc8ee85db7d9457d2248b8b92876ff13ab5baef90b WatchSource:0}: Error finding container 8dc3f58a51628a1ba1e51efc8ee85db7d9457d2248b8b92876ff13ab5baef90b: Status 404 returned error can't find the container with id 8dc3f58a51628a1ba1e51efc8ee85db7d9457d2248b8b92876ff13ab5baef90b Nov 03 13:08:55.055920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:55.055880 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" event={"ID":"0db0b8cf-03c4-4493-a3f8-464723ef7eec","Type":"ContainerStarted","Data":"8dc3f58a51628a1ba1e51efc8ee85db7d9457d2248b8b92876ff13ab5baef90b"} Nov 03 13:08:55.061983 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:55.061945 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" event={"ID":"cbfb959d-44dd-4c99-8b90-888ae2d2305e","Type":"ContainerStarted","Data":"9bd9cff847a86acdc2b3baf2775546ca0e9a5ff287b913a2b32bd95180c8b587"} Nov 03 13:08:57.559319 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:08:57.559235 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:03.160133 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.159027 4924 generic.go:356] "Generic (PLEG): container finished" podID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerID="9d10724ef3025fb868d045a1d0884e98b5d7bd293706226c9e8a2326a1f2ddce" exitCode=0 Nov 03 13:09:03.160133 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.159174 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" event={"ID":"d6690ccb-c514-4b2b-9d2d-8a580995f308","Type":"ContainerDied","Data":"9d10724ef3025fb868d045a1d0884e98b5d7bd293706226c9e8a2326a1f2ddce"} Nov 03 13:09:03.162749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.162713 4924 generic.go:356] "Generic (PLEG): container finished" podID="288531c0-4667-47fe-b92a-f556d59a0594" containerID="0e3fc1c8752d9186eee06ed763f4d07f7dd3183ddca2a858c740ad0161e5de5d" exitCode=0 Nov 03 13:09:03.162971 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.162942 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" event={"ID":"288531c0-4667-47fe-b92a-f556d59a0594","Type":"ContainerDied","Data":"0e3fc1c8752d9186eee06ed763f4d07f7dd3183ddca2a858c740ad0161e5de5d"} Nov 03 13:09:03.166890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.166859 4924 generic.go:356] "Generic (PLEG): container finished" podID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerID="dbba92d2a5e882af56fc1da2afa4b5913326b97550ac4e8f36789af8ab38181c" exitCode=0 Nov 03 13:09:03.166994 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.166965 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" event={"ID":"f8c29c94-6386-4edb-a833-07f7e3d824d1","Type":"ContainerDied","Data":"dbba92d2a5e882af56fc1da2afa4b5913326b97550ac4e8f36789af8ab38181c"} Nov 03 13:09:03.989635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.989595 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:09:03.989754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.989687 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:09:03.989754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:03.989740 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:09:04.180980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.180940 4924 generic.go:356] "Generic (PLEG): container finished" podID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerID="3e4249e2e7489317ff640669615da0818957f5d0fbd05a177158a8904632c27d" exitCode=0 Nov 03 13:09:04.181419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.181014 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" event={"ID":"c66682e7-3106-4c0f-8a27-caec24edaef2","Type":"ContainerDied","Data":"3e4249e2e7489317ff640669615da0818957f5d0fbd05a177158a8904632c27d"} Nov 03 13:09:04.186992 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.186271 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" event={"ID":"cbfb959d-44dd-4c99-8b90-888ae2d2305e","Type":"ContainerStarted","Data":"61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea"} Nov 03 13:09:04.192275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.191545 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:09:04.192275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.191630 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:09:04.192275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.191676 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:09:04.197516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.194941 4924 generic.go:356] "Generic (PLEG): container finished" podID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerID="2142625f01ad2408d496f96efcbafc6de831c02629ff5a3c7ef7c5b404d5b6e9" exitCode=0 Nov 03 13:09:04.197516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.195027 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" event={"ID":"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249","Type":"ContainerDied","Data":"2142625f01ad2408d496f96efcbafc6de831c02629ff5a3c7ef7c5b404d5b6e9"} Nov 03 13:09:04.201968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.201353 4924 generic.go:356] "Generic (PLEG): container finished" podID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerID="aa851b3116c7a1aa1188e0a9215bd0b00e89f4d8f8119c5c91c5b815bcffcf35" exitCode=0 Nov 03 13:09:04.201968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.201402 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" event={"ID":"2713157b-b9a5-4cff-a3ba-0696ed444ab9","Type":"ContainerDied","Data":"aa851b3116c7a1aa1188e0a9215bd0b00e89f4d8f8119c5c91c5b815bcffcf35"} Nov 03 13:09:04.399817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.399756 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podStartSLOduration=1.65592271 podStartE2EDuration="11.399736561s" totalImagesPullingTime="9.743813851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:04.397784338 +0000 UTC m=+277.592187934" watchObservedRunningTime="2026-11-03 13:09:04.399736561 +0000 UTC m=+277.594140155" Nov 03 13:09:04.780765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:04.780742 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:09:05.209027 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.208929 4924 generic.go:356] "Generic (PLEG): container finished" podID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerID="2a7d3183e01a7d0eb8f1ebbd173ae4cb6025f93d6626ba6f5fa02d9ed0c89800" exitCode=0 Nov 03 13:09:05.209455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.209098 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" event={"ID":"aba43d56-800d-44a3-9092-68d1a6fe20cb","Type":"ContainerDied","Data":"2a7d3183e01a7d0eb8f1ebbd173ae4cb6025f93d6626ba6f5fa02d9ed0c89800"} Nov 03 13:09:05.213179 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.213146 4924 generic.go:356] "Generic (PLEG): container finished" podID="3d512678-9504-4ce7-8479-ba09349077e8" containerID="41949e9b4dfeccad00444cf4b2ad6d5a0f0f14269b72e5fbf63b68cae947dc98" exitCode=0 Nov 03 13:09:05.213289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.213254 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" event={"ID":"3d512678-9504-4ce7-8479-ba09349077e8","Type":"ContainerDied","Data":"41949e9b4dfeccad00444cf4b2ad6d5a0f0f14269b72e5fbf63b68cae947dc98"} Nov 03 13:09:05.216551 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.216525 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" event={"ID":"0db0b8cf-03c4-4493-a3f8-464723ef7eec","Type":"ContainerStarted","Data":"0e448174e760553c9b311051a6e66d97ba04a050943bae306becca45bf12e696"} Nov 03 13:09:05.219309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.219283 4924 generic.go:356] "Generic (PLEG): container finished" podID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerID="73ffc9244e9e11b68d03169aba12f1a3eab72c09bf4385c4b079699381cf8014" exitCode=0 Nov 03 13:09:05.219410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.219370 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" event={"ID":"d6690ccb-c514-4b2b-9d2d-8a580995f308","Type":"ContainerDied","Data":"73ffc9244e9e11b68d03169aba12f1a3eab72c09bf4385c4b079699381cf8014"} Nov 03 13:09:05.223716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.223696 4924 generic.go:356] "Generic (PLEG): container finished" podID="288531c0-4667-47fe-b92a-f556d59a0594" containerID="a898294729bcebb54191cbedd136eba244c36b6f637783fcff6b24e2d02c3cb4" exitCode=0 Nov 03 13:09:05.223803 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.223771 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" event={"ID":"288531c0-4667-47fe-b92a-f556d59a0594","Type":"ContainerDied","Data":"a898294729bcebb54191cbedd136eba244c36b6f637783fcff6b24e2d02c3cb4"} Nov 03 13:09:05.758171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.757648 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" podStartSLOduration=3.116217229 podStartE2EDuration="12.757625259s" totalImagesPullingTime="9.64140803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:05.756909899 +0000 UTC m=+278.951313496" watchObservedRunningTime="2026-11-03 13:09:05.757625259 +0000 UTC m=+278.952028853" Nov 03 13:09:05.785790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:05.785767 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:09:06.123522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.123455 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:09:06.128988 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.128963 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:09:06.232296 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.232252 4924 generic.go:356] "Generic (PLEG): container finished" podID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerID="e736d40db31f6d4672a321b8619e9037688544afab1ecc5bf362c7d69b691fec" exitCode=0 Nov 03 13:09:06.232797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.232341 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" event={"ID":"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da","Type":"ContainerDied","Data":"e736d40db31f6d4672a321b8619e9037688544afab1ecc5bf362c7d69b691fec"} Nov 03 13:09:06.234398 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.234368 4924 generic.go:356] "Generic (PLEG): container finished" podID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerID="2e024533f0217bdc492e3e46e6ce1bb6e5887b48d3fb031630101a90d9218fa9" exitCode=0 Nov 03 13:09:06.234498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.234447 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" event={"ID":"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249","Type":"ContainerDied","Data":"2e024533f0217bdc492e3e46e6ce1bb6e5887b48d3fb031630101a90d9218fa9"} Nov 03 13:09:06.236603 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.236578 4924 generic.go:356] "Generic (PLEG): container finished" podID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerID="6bd0edf6cd70f0ea789baee18831df7e64ce43daff0032a81f94229c4692de90" exitCode=0 Nov 03 13:09:06.236706 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.236663 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" event={"ID":"2713157b-b9a5-4cff-a3ba-0696ed444ab9","Type":"ContainerDied","Data":"6bd0edf6cd70f0ea789baee18831df7e64ce43daff0032a81f94229c4692de90"} Nov 03 13:09:06.238780 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.238756 4924 generic.go:356] "Generic (PLEG): container finished" podID="3d512678-9504-4ce7-8479-ba09349077e8" containerID="c2644cca5fe852699c75f5e915192efec1aa8ce3295a75e9162db13aeb0835f2" exitCode=0 Nov 03 13:09:06.238881 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.238798 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" event={"ID":"3d512678-9504-4ce7-8479-ba09349077e8","Type":"ContainerDied","Data":"c2644cca5fe852699c75f5e915192efec1aa8ce3295a75e9162db13aeb0835f2"} Nov 03 13:09:06.239489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.239469 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:09:06.239635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.239541 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:09:06.240852 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.240825 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95" Nov 03 13:09:06.240947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:06.240895 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:09:07.247792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:07.247755 4924 generic.go:356] "Generic (PLEG): container finished" podID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerID="67175ab4ddc207adeb55fe9a3784ca91ad531770fd5e384a37d24d6211d84f08" exitCode=0 Nov 03 13:09:07.248317 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:07.247898 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" event={"ID":"f8c29c94-6386-4edb-a833-07f7e3d824d1","Type":"ContainerDied","Data":"67175ab4ddc207adeb55fe9a3784ca91ad531770fd5e384a37d24d6211d84f08"} Nov 03 13:09:07.252890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:07.252863 4924 generic.go:356] "Generic (PLEG): container finished" podID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerID="04d8ca751abbf26ea81df00ae4d0e981f626c542a7838de785cd30c123d603ba" exitCode=0 Nov 03 13:09:07.253160 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:07.253118 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" event={"ID":"c66682e7-3106-4c0f-8a27-caec24edaef2","Type":"ContainerDied","Data":"04d8ca751abbf26ea81df00ae4d0e981f626c542a7838de785cd30c123d603ba"} Nov 03 13:09:08.277410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:08.277286 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" event={"ID":"aba43d56-800d-44a3-9092-68d1a6fe20cb","Type":"ContainerStarted","Data":"1159fa20c3266c7a796e6815b7b95d746108a5e1b51ddadee005cbcf4f59a4cd"} Nov 03 13:09:09.297629 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:09.297590 4924 generic.go:356] "Generic (PLEG): container finished" podID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerID="1159fa20c3266c7a796e6815b7b95d746108a5e1b51ddadee005cbcf4f59a4cd" exitCode=0 Nov 03 13:09:09.298146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:09.297663 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" event={"ID":"aba43d56-800d-44a3-9092-68d1a6fe20cb","Type":"ContainerDied","Data":"1159fa20c3266c7a796e6815b7b95d746108a5e1b51ddadee005cbcf4f59a4cd"} Nov 03 13:09:15.368756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:15.368721 4924 generic.go:356] "Generic (PLEG): container finished" podID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerID="0cee02d641a8f3ace8f3d8f53c9b6c0d5c6764945efc494f93fef01c5b1d5dc8" exitCode=0 Nov 03 13:09:15.369327 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:15.368768 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" event={"ID":"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da","Type":"ContainerDied","Data":"0cee02d641a8f3ace8f3d8f53c9b6c0d5c6764945efc494f93fef01c5b1d5dc8"} Nov 03 13:09:18.206786 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:18.206736 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:18.442873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:18.442832 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:18.585961 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:09:18.585751 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:09:23.477180 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:23.476263 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" event={"ID":"d6690ccb-c514-4b2b-9d2d-8a580995f308","Type":"ContainerStarted","Data":"ddfb0bd89546045513275770b5c87e0b018a8980081b88b67300ad86c1de6768"} Nov 03 13:09:23.484035 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:23.483795 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" event={"ID":"288531c0-4667-47fe-b92a-f556d59a0594","Type":"ContainerStarted","Data":"032cce9008bc6687fe20c4b111af92333c4103563990890fe9f4e65cd1491763"} Nov 03 13:09:23.588847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:23.588728 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" podStartSLOduration=3.218073206 podStartE2EDuration="37.588714516s" totalImagesPullingTime="34.37064131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:23.588199878 +0000 UTC m=+296.782603473" watchObservedRunningTime="2026-11-03 13:09:23.588714516 +0000 UTC m=+296.783118109" Nov 03 13:09:23.700796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:23.699432 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" podStartSLOduration=2.366200079 podStartE2EDuration="33.699411494s" totalImagesPullingTime="31.333211415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:23.697530799 +0000 UTC m=+296.891934412" watchObservedRunningTime="2026-11-03 13:09:23.699411494 +0000 UTC m=+296.893815090" Nov 03 13:09:24.495408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:24.494879 4924 generic.go:356] "Generic (PLEG): container finished" podID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerID="ddfb0bd89546045513275770b5c87e0b018a8980081b88b67300ad86c1de6768" exitCode=0 Nov 03 13:09:24.495408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:24.494991 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" event={"ID":"d6690ccb-c514-4b2b-9d2d-8a580995f308","Type":"ContainerDied","Data":"ddfb0bd89546045513275770b5c87e0b018a8980081b88b67300ad86c1de6768"} Nov 03 13:09:24.500015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:24.499969 4924 generic.go:356] "Generic (PLEG): container finished" podID="288531c0-4667-47fe-b92a-f556d59a0594" containerID="032cce9008bc6687fe20c4b111af92333c4103563990890fe9f4e65cd1491763" exitCode=0 Nov 03 13:09:24.500193 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:24.500031 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" event={"ID":"288531c0-4667-47fe-b92a-f556d59a0594","Type":"ContainerDied","Data":"032cce9008bc6687fe20c4b111af92333c4103563990890fe9f4e65cd1491763"} Nov 03 13:09:25.514493 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.514460 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" event={"ID":"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249","Type":"ContainerStarted","Data":"317315936a279ef9f0df4d566cd39e5d837499b1643d8da01a1e10ac1aa8374f"} Nov 03 13:09:25.517582 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.517526 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" event={"ID":"2713157b-b9a5-4cff-a3ba-0696ed444ab9","Type":"ContainerStarted","Data":"435507c6cbe21cbbb714e58ecf811f42d4309fa049d91023337153ab932450ef"} Nov 03 13:09:25.521120 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.521080 4924 generic.go:356] "Generic (PLEG): container finished" podID="3d512678-9504-4ce7-8479-ba09349077e8" containerID="92b0b9fba0ce31c11259e1392749053ef7476582dc567b5dddd7ef21f9a9e694" exitCode=0 Nov 03 13:09:25.521236 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.521218 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" event={"ID":"3d512678-9504-4ce7-8479-ba09349077e8","Type":"ContainerDied","Data":"92b0b9fba0ce31c11259e1392749053ef7476582dc567b5dddd7ef21f9a9e694"} Nov 03 13:09:25.525042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.525017 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" event={"ID":"c66682e7-3106-4c0f-8a27-caec24edaef2","Type":"ContainerStarted","Data":"1dc6c4a37e19815480352755ddf764a03227ea13e2cc52563d148a5cc17d9fb4"} Nov 03 13:09:25.684322 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.684291 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:09:25.739888 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.739817 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:09:25.740383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.740334 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" podStartSLOduration=3.6606045529999998 podStartE2EDuration="35.740308826s" totalImagesPullingTime="32.079704273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:25.739466419 +0000 UTC m=+298.933870014" watchObservedRunningTime="2026-11-03 13:09:25.740308826 +0000 UTC m=+298.934712435" Nov 03 13:09:25.741040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.740997 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" podStartSLOduration=2.882400535 podStartE2EDuration="35.740983886s" totalImagesPullingTime="32.858583351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:25.632167207 +0000 UTC m=+298.826570854" watchObservedRunningTime="2026-11-03 13:09:25.740983886 +0000 UTC m=+298.935387482" Nov 03 13:09:25.789345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.789177 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-util\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-util\") pod \"d6690ccb-c514-4b2b-9d2d-8a580995f308\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " Nov 03 13:09:25.789345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.789296 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6690ccb-c514-4b2b-9d2d-8a580995f308-kube-api-access-5zxf5\" (UniqueName: \"kubernetes.io/projected/d6690ccb-c514-4b2b-9d2d-8a580995f308-kube-api-access-5zxf5\") pod \"d6690ccb-c514-4b2b-9d2d-8a580995f308\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " Nov 03 13:09:25.789345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.789342 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-bundle\") pod \"d6690ccb-c514-4b2b-9d2d-8a580995f308\" (UID: \"d6690ccb-c514-4b2b-9d2d-8a580995f308\") " Nov 03 13:09:25.790100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.790053 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-bundle" pod "d6690ccb-c514-4b2b-9d2d-8a580995f308" (UID: "d6690ccb-c514-4b2b-9d2d-8a580995f308"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:25.792012 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.791984 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6690ccb-c514-4b2b-9d2d-8a580995f308-kube-api-access-5zxf5" pod "d6690ccb-c514-4b2b-9d2d-8a580995f308" (UID: "d6690ccb-c514-4b2b-9d2d-8a580995f308"). InnerVolumeSpecName "kube-api-access-5zxf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:25.796659 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.796631 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-util" pod "d6690ccb-c514-4b2b-9d2d-8a580995f308" (UID: "d6690ccb-c514-4b2b-9d2d-8a580995f308"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:25.854168 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.854081 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" podStartSLOduration=3.895716485 podStartE2EDuration="35.854053482s" totalImagesPullingTime="31.958336997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:25.852399661 +0000 UTC m=+299.046803255" watchObservedRunningTime="2026-11-03 13:09:25.854053482 +0000 UTC m=+299.048457077" Nov 03 13:09:25.889927 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.889901 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-util\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-util\") pod \"288531c0-4667-47fe-b92a-f556d59a0594\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " Nov 03 13:09:25.890059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.889938 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288531c0-4667-47fe-b92a-f556d59a0594-kube-api-access-dkjkg\" (UniqueName: \"kubernetes.io/projected/288531c0-4667-47fe-b92a-f556d59a0594-kube-api-access-dkjkg\") pod \"288531c0-4667-47fe-b92a-f556d59a0594\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " Nov 03 13:09:25.890059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.890012 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-bundle\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-bundle\") pod \"288531c0-4667-47fe-b92a-f556d59a0594\" (UID: \"288531c0-4667-47fe-b92a-f556d59a0594\") " Nov 03 13:09:25.890358 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.890335 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:25.890464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.890363 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6690ccb-c514-4b2b-9d2d-8a580995f308-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:25.890464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.890388 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zxf5\" (UniqueName: \"kubernetes.io/projected/d6690ccb-c514-4b2b-9d2d-8a580995f308-kube-api-access-5zxf5\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:25.890792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.890767 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-bundle" pod "288531c0-4667-47fe-b92a-f556d59a0594" (UID: "288531c0-4667-47fe-b92a-f556d59a0594"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:25.895214 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.895179 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-util" pod "288531c0-4667-47fe-b92a-f556d59a0594" (UID: "288531c0-4667-47fe-b92a-f556d59a0594"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:25.897593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.897565 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288531c0-4667-47fe-b92a-f556d59a0594-kube-api-access-dkjkg" pod "288531c0-4667-47fe-b92a-f556d59a0594" (UID: "288531c0-4667-47fe-b92a-f556d59a0594"). InnerVolumeSpecName "kube-api-access-dkjkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:25.990920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.990889 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:25.990920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.990918 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkjkg\" (UniqueName: \"kubernetes.io/projected/288531c0-4667-47fe-b92a-f556d59a0594-kube-api-access-dkjkg\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:25.991110 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:25.990928 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/288531c0-4667-47fe-b92a-f556d59a0594-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:26.532895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.532808 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" Nov 03 13:09:26.532895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.532826 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717lxjd" event={"ID":"d6690ccb-c514-4b2b-9d2d-8a580995f308","Type":"ContainerDied","Data":"96abac70b9487302fce6ab9090bd93934f080951ab5791dd2edd2e674b958884"} Nov 03 13:09:26.532895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.532877 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96abac70b9487302fce6ab9090bd93934f080951ab5791dd2edd2e674b958884" Nov 03 13:09:26.538162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.538132 4924 generic.go:356] "Generic (PLEG): container finished" podID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerID="1dc6c4a37e19815480352755ddf764a03227ea13e2cc52563d148a5cc17d9fb4" exitCode=0 Nov 03 13:09:26.538315 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.538218 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" event={"ID":"c66682e7-3106-4c0f-8a27-caec24edaef2","Type":"ContainerDied","Data":"1dc6c4a37e19815480352755ddf764a03227ea13e2cc52563d148a5cc17d9fb4"} Nov 03 13:09:26.540394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.540371 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" Nov 03 13:09:26.540515 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.540397 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6h9bk6" event={"ID":"288531c0-4667-47fe-b92a-f556d59a0594","Type":"ContainerDied","Data":"38dddd4100b7b3c93d2c96ee84c65460d2c721f0a951abc4254b64b49540f30f"} Nov 03 13:09:26.540515 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.540444 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38dddd4100b7b3c93d2c96ee84c65460d2c721f0a951abc4254b64b49540f30f" Nov 03 13:09:26.545518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.545489 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" event={"ID":"f8c29c94-6386-4edb-a833-07f7e3d824d1","Type":"ContainerStarted","Data":"84c096eb31fbb7976199a72fb44142e315c426922e95d3dab050fd5a5ea4b25b"} Nov 03 13:09:26.547973 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.547947 4924 generic.go:356] "Generic (PLEG): container finished" podID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerID="ade2764b06c784c0bce1f210bbef0e7b9e88fa6189c6345f684b0fe33e569739" exitCode=0 Nov 03 13:09:26.548063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.548026 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" event={"ID":"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da","Type":"ContainerDied","Data":"ade2764b06c784c0bce1f210bbef0e7b9e88fa6189c6345f684b0fe33e569739"} Nov 03 13:09:26.550445 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.550342 4924 generic.go:356] "Generic (PLEG): container finished" podID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerID="317315936a279ef9f0df4d566cd39e5d837499b1643d8da01a1e10ac1aa8374f" exitCode=0 Nov 03 13:09:26.550445 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.550424 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" event={"ID":"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249","Type":"ContainerDied","Data":"317315936a279ef9f0df4d566cd39e5d837499b1643d8da01a1e10ac1aa8374f"} Nov 03 13:09:26.552706 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.552680 4924 generic.go:356] "Generic (PLEG): container finished" podID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerID="435507c6cbe21cbbb714e58ecf811f42d4309fa049d91023337153ab932450ef" exitCode=0 Nov 03 13:09:26.552857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.552743 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" event={"ID":"2713157b-b9a5-4cff-a3ba-0696ed444ab9","Type":"ContainerDied","Data":"435507c6cbe21cbbb714e58ecf811f42d4309fa049d91023337153ab932450ef"} Nov 03 13:09:26.684537 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.684510 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:09:26.754643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.754578 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" podStartSLOduration=3.4163391020000002 podStartE2EDuration="40.75455805s" totalImagesPullingTime="37.338218948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:26.753576036 +0000 UTC m=+299.947979632" watchObservedRunningTime="2026-11-03 13:09:26.75455805 +0000 UTC m=+299.948961646" Nov 03 13:09:26.799795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.799716 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-bundle\") pod \"3d512678-9504-4ce7-8479-ba09349077e8\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " Nov 03 13:09:26.799795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.799780 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-util\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-util\") pod \"3d512678-9504-4ce7-8479-ba09349077e8\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " Nov 03 13:09:26.800023 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.799877 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d512678-9504-4ce7-8479-ba09349077e8-kube-api-access-tjb8x\" (UniqueName: \"kubernetes.io/projected/3d512678-9504-4ce7-8479-ba09349077e8-kube-api-access-tjb8x\") pod \"3d512678-9504-4ce7-8479-ba09349077e8\" (UID: \"3d512678-9504-4ce7-8479-ba09349077e8\") " Nov 03 13:09:26.800467 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.800436 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-bundle" pod "3d512678-9504-4ce7-8479-ba09349077e8" (UID: "3d512678-9504-4ce7-8479-ba09349077e8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:26.802644 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.802606 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d512678-9504-4ce7-8479-ba09349077e8-kube-api-access-tjb8x" pod "3d512678-9504-4ce7-8479-ba09349077e8" (UID: "3d512678-9504-4ce7-8479-ba09349077e8"). InnerVolumeSpecName "kube-api-access-tjb8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:26.805951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.805930 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-util" pod "3d512678-9504-4ce7-8479-ba09349077e8" (UID: "3d512678-9504-4ce7-8479-ba09349077e8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:26.904525 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.901872 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjb8x\" (UniqueName: \"kubernetes.io/projected/3d512678-9504-4ce7-8479-ba09349077e8-kube-api-access-tjb8x\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:26.904525 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.901910 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:26.904525 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.901933 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d512678-9504-4ce7-8479-ba09349077e8-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:26.973119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.973059 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:09:26.973258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:26.973062 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:09:27.559143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.559108 4924 generic.go:356] "Generic (PLEG): container finished" podID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerID="84c096eb31fbb7976199a72fb44142e315c426922e95d3dab050fd5a5ea4b25b" exitCode=0 Nov 03 13:09:27.559591 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.559168 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" event={"ID":"f8c29c94-6386-4edb-a833-07f7e3d824d1","Type":"ContainerDied","Data":"84c096eb31fbb7976199a72fb44142e315c426922e95d3dab050fd5a5ea4b25b"} Nov 03 13:09:27.561127 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.561104 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" event={"ID":"3d512678-9504-4ce7-8479-ba09349077e8","Type":"ContainerDied","Data":"f430264d414d2bfe159de3f643beecd99670424c69c805197e0659b507831d32"} Nov 03 13:09:27.561239 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.561129 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xn586" Nov 03 13:09:27.561239 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.561144 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f430264d414d2bfe159de3f643beecd99670424c69c805197e0659b507831d32" Nov 03 13:09:27.563472 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.563447 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" event={"ID":"aba43d56-800d-44a3-9092-68d1a6fe20cb","Type":"ContainerStarted","Data":"55306bb5a1866474b2a22dd73c353a3d25d0797efcf3dad933e03b719844fd11"} Nov 03 13:09:27.725277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.725243 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:09:27.751884 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.751858 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:09:27.783654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.783634 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:09:27.786513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.786495 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:09:27.809899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.809835 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-bundle\") pod \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " Nov 03 13:09:27.809899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.809886 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-util\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-util\") pod \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " Nov 03 13:09:27.810056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.809922 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2713157b-b9a5-4cff-a3ba-0696ed444ab9-kube-api-access-p7m9z\" (UniqueName: \"kubernetes.io/projected/2713157b-b9a5-4cff-a3ba-0696ed444ab9-kube-api-access-p7m9z\") pod \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " Nov 03 13:09:27.810056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.809957 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-bundle\") pod \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " Nov 03 13:09:27.810056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.809997 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-util\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-util\") pod \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " Nov 03 13:09:27.810056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810035 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-util\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-util\") pod \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " Nov 03 13:09:27.810284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810062 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66682e7-3106-4c0f-8a27-caec24edaef2-kube-api-access-z5k7r\" (UniqueName: \"kubernetes.io/projected/c66682e7-3106-4c0f-8a27-caec24edaef2-kube-api-access-z5k7r\") pod \"c66682e7-3106-4c0f-8a27-caec24edaef2\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " Nov 03 13:09:27.810284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810113 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-kube-api-access-vnwwc\" (UniqueName: \"kubernetes.io/projected/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-kube-api-access-vnwwc\") pod \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\" (UID: \"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da\") " Nov 03 13:09:27.810284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810177 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-bundle\") pod \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\" (UID: \"2713157b-b9a5-4cff-a3ba-0696ed444ab9\") " Nov 03 13:09:27.810284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810219 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-bundle\") pod \"c66682e7-3106-4c0f-8a27-caec24edaef2\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " Nov 03 13:09:27.810284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810240 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-util\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-util\") pod \"c66682e7-3106-4c0f-8a27-caec24edaef2\" (UID: \"c66682e7-3106-4c0f-8a27-caec24edaef2\") " Nov 03 13:09:27.810284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810262 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-kube-api-access-7tm87\" (UniqueName: \"kubernetes.io/projected/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-kube-api-access-7tm87\") pod \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\" (UID: \"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249\") " Nov 03 13:09:27.810584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810437 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-bundle" pod "c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" (UID: "c7e0ec10-ffe1-426d-a0a2-0c902d35c6da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:27.810584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810540 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.810682 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.810566 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-bundle" pod "bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" (UID: "bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:27.811489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.811292 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-bundle" pod "c66682e7-3106-4c0f-8a27-caec24edaef2" (UID: "c66682e7-3106-4c0f-8a27-caec24edaef2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:27.811489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.811430 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-bundle" pod "2713157b-b9a5-4cff-a3ba-0696ed444ab9" (UID: "2713157b-b9a5-4cff-a3ba-0696ed444ab9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:27.813201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.813147 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-kube-api-access-7tm87" pod "bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" (UID: "bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249"). InnerVolumeSpecName "kube-api-access-7tm87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:27.813430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.813409 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66682e7-3106-4c0f-8a27-caec24edaef2-kube-api-access-z5k7r" pod "c66682e7-3106-4c0f-8a27-caec24edaef2" (UID: "c66682e7-3106-4c0f-8a27-caec24edaef2"). InnerVolumeSpecName "kube-api-access-z5k7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:27.814116 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.814086 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-kube-api-access-vnwwc" pod "c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" (UID: "c7e0ec10-ffe1-426d-a0a2-0c902d35c6da"). InnerVolumeSpecName "kube-api-access-vnwwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:27.815547 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.815523 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2713157b-b9a5-4cff-a3ba-0696ed444ab9-kube-api-access-p7m9z" pod "2713157b-b9a5-4cff-a3ba-0696ed444ab9" (UID: "2713157b-b9a5-4cff-a3ba-0696ed444ab9"). InnerVolumeSpecName "kube-api-access-p7m9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:27.818285 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.818135 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-util" pod "bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" (UID: "bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:27.819153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.819130 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-util" pod "c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" (UID: "c7e0ec10-ffe1-426d-a0a2-0c902d35c6da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:27.819222 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.819181 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-util" pod "c66682e7-3106-4c0f-8a27-caec24edaef2" (UID: "c66682e7-3106-4c0f-8a27-caec24edaef2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:27.819570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.819552 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-util" pod "2713157b-b9a5-4cff-a3ba-0696ed444ab9" (UID: "2713157b-b9a5-4cff-a3ba-0696ed444ab9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:27.825380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.825336 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" podStartSLOduration=3.76007625 podStartE2EDuration="35.82532518s" totalImagesPullingTime="32.06524893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:27.824226246 +0000 UTC m=+301.018629836" watchObservedRunningTime="2026-11-03 13:09:27.82532518 +0000 UTC m=+301.019728764" Nov 03 13:09:27.911571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911535 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5k7r\" (UniqueName: \"kubernetes.io/projected/c66682e7-3106-4c0f-8a27-caec24edaef2-kube-api-access-z5k7r\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911566 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnwwc\" (UniqueName: \"kubernetes.io/projected/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-kube-api-access-vnwwc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911576 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911586 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911595 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c66682e7-3106-4c0f-8a27-caec24edaef2-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911604 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tm87\" (UniqueName: \"kubernetes.io/projected/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-kube-api-access-7tm87\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911614 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911622 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e0ec10-ffe1-426d-a0a2-0c902d35c6da-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911630 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7m9z\" (UniqueName: \"kubernetes.io/projected/2713157b-b9a5-4cff-a3ba-0696ed444ab9-kube-api-access-p7m9z\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911637 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:27.911836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:27.911645 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2713157b-b9a5-4cff-a3ba-0696ed444ab9-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:28.570054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.570013 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" event={"ID":"c7e0ec10-ffe1-426d-a0a2-0c902d35c6da","Type":"ContainerDied","Data":"8429a6d0e079a80af84837002d749a3727058c6b4f3fd6bc06acfb3527fb9d3f"} Nov 03 13:09:28.570549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.570065 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8429a6d0e079a80af84837002d749a3727058c6b4f3fd6bc06acfb3527fb9d3f" Nov 03 13:09:28.570549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.570024 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644n9nzx" Nov 03 13:09:28.571934 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.571906 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" event={"ID":"bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249","Type":"ContainerDied","Data":"9afb1779f34350e67c0bb90204027cdbe9cfdf77997575c00ff13e9f03c04551"} Nov 03 13:09:28.572062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.571951 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9afb1779f34350e67c0bb90204027cdbe9cfdf77997575c00ff13e9f03c04551" Nov 03 13:09:28.572062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.571912 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j6hkl" Nov 03 13:09:28.574008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.573982 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" Nov 03 13:09:28.574469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.574338 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adhqgg" event={"ID":"2713157b-b9a5-4cff-a3ba-0696ed444ab9","Type":"ContainerDied","Data":"5586bd70cd12196c3e9d372a63b1f347cc6fc1ce1db7dc2463f2ea99a5198ca8"} Nov 03 13:09:28.574580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.574488 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5586bd70cd12196c3e9d372a63b1f347cc6fc1ce1db7dc2463f2ea99a5198ca8" Nov 03 13:09:28.584203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.583947 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" event={"ID":"c66682e7-3106-4c0f-8a27-caec24edaef2","Type":"ContainerDied","Data":"e2e517ac90007a83b3b9574cc25d68200f7d93f9822a103f4b80c5c272d1f778"} Nov 03 13:09:28.584203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.584031 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e517ac90007a83b3b9574cc25d68200f7d93f9822a103f4b80c5c272d1f778" Nov 03 13:09:28.584203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.583966 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2k2jt" Nov 03 13:09:28.586403 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.586381 4924 generic.go:356] "Generic (PLEG): container finished" podID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerID="55306bb5a1866474b2a22dd73c353a3d25d0797efcf3dad933e03b719844fd11" exitCode=0 Nov 03 13:09:28.586495 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.586422 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" event={"ID":"aba43d56-800d-44a3-9092-68d1a6fe20cb","Type":"ContainerDied","Data":"55306bb5a1866474b2a22dd73c353a3d25d0797efcf3dad933e03b719844fd11"} Nov 03 13:09:28.700652 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.700626 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:09:28.719273 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.719247 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-bundle\") pod \"f8c29c94-6386-4edb-a833-07f7e3d824d1\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " Nov 03 13:09:28.719397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.719315 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c29c94-6386-4edb-a833-07f7e3d824d1-kube-api-access-fg96s\" (UniqueName: \"kubernetes.io/projected/f8c29c94-6386-4edb-a833-07f7e3d824d1-kube-api-access-fg96s\") pod \"f8c29c94-6386-4edb-a833-07f7e3d824d1\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " Nov 03 13:09:28.719397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.719369 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-util\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-util\") pod \"f8c29c94-6386-4edb-a833-07f7e3d824d1\" (UID: \"f8c29c94-6386-4edb-a833-07f7e3d824d1\") " Nov 03 13:09:28.719978 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.719956 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-bundle" pod "f8c29c94-6386-4edb-a833-07f7e3d824d1" (UID: "f8c29c94-6386-4edb-a833-07f7e3d824d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:28.721473 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.721446 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c29c94-6386-4edb-a833-07f7e3d824d1-kube-api-access-fg96s" pod "f8c29c94-6386-4edb-a833-07f7e3d824d1" (UID: "f8c29c94-6386-4edb-a833-07f7e3d824d1"). InnerVolumeSpecName "kube-api-access-fg96s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:28.725032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.725009 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-util" pod "f8c29c94-6386-4edb-a833-07f7e3d824d1" (UID: "f8c29c94-6386-4edb-a833-07f7e3d824d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:28.819980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.819945 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:28.819980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.819973 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg96s\" (UniqueName: \"kubernetes.io/projected/f8c29c94-6386-4edb-a833-07f7e3d824d1-kube-api-access-fg96s\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:28.819980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:28.819983 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8c29c94-6386-4edb-a833-07f7e3d824d1-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:29.594456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.594426 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" Nov 03 13:09:29.594456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.594436 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmd6jl" event={"ID":"f8c29c94-6386-4edb-a833-07f7e3d824d1","Type":"ContainerDied","Data":"6b79ea5497bae6e27b7c82860789db2ebb4f094c230eab1fb71a913ea7ddec4e"} Nov 03 13:09:29.594885 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.594489 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b79ea5497bae6e27b7c82860789db2ebb4f094c230eab1fb71a913ea7ddec4e" Nov 03 13:09:29.704534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.704460 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:29.720369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.720344 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:09:29.727271 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.727245 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba43d56-800d-44a3-9092-68d1a6fe20cb-kube-api-access-p5q79\" (UniqueName: \"kubernetes.io/projected/aba43d56-800d-44a3-9092-68d1a6fe20cb-kube-api-access-p5q79\") pod \"aba43d56-800d-44a3-9092-68d1a6fe20cb\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " Nov 03 13:09:29.727399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.727296 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-bundle\") pod \"aba43d56-800d-44a3-9092-68d1a6fe20cb\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " Nov 03 13:09:29.727399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.727318 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-util\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-util\") pod \"aba43d56-800d-44a3-9092-68d1a6fe20cb\" (UID: \"aba43d56-800d-44a3-9092-68d1a6fe20cb\") " Nov 03 13:09:29.728112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.728060 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-bundle" pod "aba43d56-800d-44a3-9092-68d1a6fe20cb" (UID: "aba43d56-800d-44a3-9092-68d1a6fe20cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:29.729531 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.729503 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba43d56-800d-44a3-9092-68d1a6fe20cb-kube-api-access-p5q79" pod "aba43d56-800d-44a3-9092-68d1a6fe20cb" (UID: "aba43d56-800d-44a3-9092-68d1a6fe20cb"). InnerVolumeSpecName "kube-api-access-p5q79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:29.735395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.735369 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-util" pod "aba43d56-800d-44a3-9092-68d1a6fe20cb" (UID: "aba43d56-800d-44a3-9092-68d1a6fe20cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:29.828040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.828010 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5q79\" (UniqueName: \"kubernetes.io/projected/aba43d56-800d-44a3-9092-68d1a6fe20cb-kube-api-access-p5q79\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:29.828040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.828038 4924 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-bundle\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:29.828040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.828047 4924 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aba43d56-800d-44a3-9092-68d1a6fe20cb-util\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:29.943774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:29.943739 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:30.331195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.331116 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:30.537085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537043 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g"] Nov 03 13:09:30.537376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537363 4924 cpu_manager.go:414] "containerMap: removing container" podUID="3d512678-9504-4ce7-8479-ba09349077e8" containerName="pull" Nov 03 13:09:30.537417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537377 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d512678-9504-4ce7-8479-ba09349077e8" containerName="pull" Nov 03 13:09:30.537417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537387 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerName="util" Nov 03 13:09:30.537417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537392 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerName="util" Nov 03 13:09:30.537417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537399 4924 cpu_manager.go:414] "containerMap: removing container" podUID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerName="pull" Nov 03 13:09:30.537417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537404 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerName="pull" Nov 03 13:09:30.537417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537409 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerName="pull" Nov 03 13:09:30.537417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537414 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537421 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537427 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537433 4924 cpu_manager.go:414] "containerMap: removing container" podUID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537438 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537450 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537455 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537462 4924 cpu_manager.go:414] "containerMap: removing container" podUID="288531c0-4667-47fe-b92a-f556d59a0594" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537466 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288531c0-4667-47fe-b92a-f556d59a0594" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537472 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537476 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537481 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537485 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537491 4924 cpu_manager.go:414] "containerMap: removing container" podUID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537495 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537503 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537508 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537513 4924 cpu_manager.go:414] "containerMap: removing container" podUID="288531c0-4667-47fe-b92a-f556d59a0594" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537517 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288531c0-4667-47fe-b92a-f556d59a0594" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537525 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537530 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537537 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537541 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537547 4924 cpu_manager.go:414] "containerMap: removing container" podUID="3d512678-9504-4ce7-8479-ba09349077e8" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537551 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d512678-9504-4ce7-8479-ba09349077e8" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537557 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537561 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerName="pull" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537568 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537573 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537579 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537584 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerName="extract" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537591 4924 cpu_manager.go:414] "containerMap: removing container" podUID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537595 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537603 4924 cpu_manager.go:414] "containerMap: removing container" podUID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537609 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537618 4924 cpu_manager.go:414] "containerMap: removing container" podUID="288531c0-4667-47fe-b92a-f556d59a0594" containerName="util" Nov 03 13:09:30.537608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537622 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288531c0-4667-47fe-b92a-f556d59a0594" containerName="util" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537628 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerName="pull" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537632 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerName="pull" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537637 4924 cpu_manager.go:414] "containerMap: removing container" podUID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerName="pull" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537641 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerName="pull" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537648 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537652 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537658 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537663 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537668 4924 cpu_manager.go:414] "containerMap: removing container" podUID="3d512678-9504-4ce7-8479-ba09349077e8" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537672 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d512678-9504-4ce7-8479-ba09349077e8" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537719 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2713157b-b9a5-4cff-a3ba-0696ed444ab9" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537726 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8ccbfa-d9a5-4989-b3b2-1992bd5f6249" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537733 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="288531c0-4667-47fe-b92a-f556d59a0594" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537740 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66682e7-3106-4c0f-8a27-caec24edaef2" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537746 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c29c94-6386-4edb-a833-07f7e3d824d1" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537752 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6690ccb-c514-4b2b-9d2d-8a580995f308" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537758 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d512678-9504-4ce7-8479-ba09349077e8" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537764 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e0ec10-ffe1-426d-a0a2-0c902d35c6da" containerName="extract" Nov 03 13:09:30.538579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.537769 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba43d56-800d-44a3-9092-68d1a6fe20cb" containerName="extract" Nov 03 13:09:30.541055 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.541040 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.600552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.600475 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 13:09:30.601361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.601336 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" event={"ID":"aba43d56-800d-44a3-9092-68d1a6fe20cb","Type":"ContainerDied","Data":"e050c0dac947c7ce33d5d37a940e2080a566f91ff30c7a2866c4dc9de4f72efd"} Nov 03 13:09:30.601480 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.601381 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e050c0dac947c7ce33d5d37a940e2080a566f91ff30c7a2866c4dc9de4f72efd" Nov 03 13:09:30.601480 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.601383 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nwxk" Nov 03 13:09:30.604532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.604517 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 13:09:30.608727 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.608705 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 13:09:30.608828 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.608756 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-clsch\"" Nov 03 13:09:30.608828 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.608772 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 13:09:30.608925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.608869 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 13:09:30.634337 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.634308 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.634447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.634340 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff69b0d-de1c-43db-a98f-23a6abe6841b-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.634447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.634373 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zl7\" (UniqueName: \"kubernetes.io/projected/cff69b0d-de1c-43db-a98f-23a6abe6841b-kube-api-access-74zl7\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.634527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.634509 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.634617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.634602 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.634662 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.634642 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff69b0d-de1c-43db-a98f-23a6abe6841b-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.697870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.697839 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g"] Nov 03 13:09:30.735878 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.735844 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zl7\" (UniqueName: \"kubernetes.io/projected/cff69b0d-de1c-43db-a98f-23a6abe6841b-kube-api-access-74zl7\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.736015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.735885 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.736015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.735924 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.736015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.735954 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff69b0d-de1c-43db-a98f-23a6abe6841b-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.736015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.735975 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.736015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.735992 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff69b0d-de1c-43db-a98f-23a6abe6841b-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.736426 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.736399 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cff69b0d-de1c-43db-a98f-23a6abe6841b-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.738392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.738355 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cff69b0d-de1c-43db-a98f-23a6abe6841b-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.738547 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.738528 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.738694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.738601 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.738694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.738625 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cff69b0d-de1c-43db-a98f-23a6abe6841b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.769232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.769197 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:30.793642 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.793618 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zl7\" (UniqueName: \"kubernetes.io/projected/cff69b0d-de1c-43db-a98f-23a6abe6841b-kube-api-access-74zl7\") pod \"opendatahub-operator-controller-manager-5b7574c576-rf94g\" (UID: \"cff69b0d-de1c-43db-a98f-23a6abe6841b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.857181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.857110 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:30.919015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:30.918979 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:31.038758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:31.038728 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:31.186476 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:31.186443 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:31.363520 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:31.363491 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g"] Nov 03 13:09:31.364509 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:09:31.364483 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff69b0d_de1c_43db_a98f_23a6abe6841b.slice/crio-4108f4342ba9654a01525bc649e7d60f7c7e4b88dfa664cb2ee3feba1bb3ccbf WatchSource:0}: Error finding container 4108f4342ba9654a01525bc649e7d60f7c7e4b88dfa664cb2ee3feba1bb3ccbf: Status 404 returned error can't find the container with id 4108f4342ba9654a01525bc649e7d60f7c7e4b88dfa664cb2ee3feba1bb3ccbf Nov 03 13:09:31.366218 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:31.366200 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:09:31.607330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:31.607239 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" event={"ID":"cff69b0d-de1c-43db-a98f-23a6abe6841b","Type":"ContainerStarted","Data":"4108f4342ba9654a01525bc649e7d60f7c7e4b88dfa664cb2ee3feba1bb3ccbf"} Nov 03 13:09:33.288442 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:33.288370 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:33.430749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:33.430715 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:33.872602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:33.872569 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:34.010174 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:34.010144 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:35.629606 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:35.629570 4924 generic.go:356] "Generic (PLEG): container finished" podID="cff69b0d-de1c-43db-a98f-23a6abe6841b" containerID="e000ce6dcb41df56676058372d1b498c0c09d9af2fc3eff6c5febafa15de35cf" exitCode=0 Nov 03 13:09:35.629985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:35.629630 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" event={"ID":"cff69b0d-de1c-43db-a98f-23a6abe6841b","Type":"ContainerDied","Data":"e000ce6dcb41df56676058372d1b498c0c09d9af2fc3eff6c5febafa15de35cf"} Nov 03 13:09:36.635968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:36.635932 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" event={"ID":"cff69b0d-de1c-43db-a98f-23a6abe6841b","Type":"ContainerStarted","Data":"5101a9c6a9f58f3d6bbe784bd86133917953bca2799c92a6d17a989d12ea063b"} Nov 03 13:09:36.636490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:36.636025 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:36.857638 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:36.857534 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" podStartSLOduration=1.984185809 podStartE2EDuration="6.857520152s" totalImagesPullingTime="4.380054395s" totalInitContainerRuntime="493.279948ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:36.811855026 +0000 UTC m=+310.006258624" watchObservedRunningTime="2026-11-03 13:09:36.857520152 +0000 UTC m=+310.051923747" Nov 03 13:09:39.475184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:39.475144 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:39.615993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:39.615964 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:41.908531 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:41.908499 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:44.936640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:44.936607 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2"] Nov 03 13:09:44.942148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:44.942129 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" Nov 03 13:09:45.037261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.037223 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:09:45.037261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.037248 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xxh4b\"" Nov 03 13:09:45.037432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.037266 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 13:09:45.039251 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.039225 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2"] Nov 03 13:09:45.054705 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.054677 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d85c2\" (UniqueName: \"kubernetes.io/projected/6f5d995f-6ee8-496b-80d6-1ad8c44689ec-kube-api-access-d85c2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2\" (UID: \"6f5d995f-6ee8-496b-80d6-1ad8c44689ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" Nov 03 13:09:45.054824 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.054805 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f5d995f-6ee8-496b-80d6-1ad8c44689ec-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2\" (UID: \"6f5d995f-6ee8-496b-80d6-1ad8c44689ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" Nov 03 13:09:45.155944 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.155912 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f5d995f-6ee8-496b-80d6-1ad8c44689ec-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2\" (UID: \"6f5d995f-6ee8-496b-80d6-1ad8c44689ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" Nov 03 13:09:45.155944 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.155956 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d85c2\" (UniqueName: \"kubernetes.io/projected/6f5d995f-6ee8-496b-80d6-1ad8c44689ec-kube-api-access-d85c2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2\" (UID: \"6f5d995f-6ee8-496b-80d6-1ad8c44689ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" Nov 03 13:09:45.156414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.156393 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f5d995f-6ee8-496b-80d6-1ad8c44689ec-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2\" (UID: \"6f5d995f-6ee8-496b-80d6-1ad8c44689ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" Nov 03 13:09:45.215050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.214982 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d85c2\" (UniqueName: \"kubernetes.io/projected/6f5d995f-6ee8-496b-80d6-1ad8c44689ec-kube-api-access-d85c2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2\" (UID: \"6f5d995f-6ee8-496b-80d6-1ad8c44689ec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" Nov 03 13:09:45.254048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.254015 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" Nov 03 13:09:45.725489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:45.725465 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2"] Nov 03 13:09:45.727601 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:09:45.727573 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5d995f_6ee8_496b_80d6_1ad8c44689ec.slice/crio-029b7dcfe19a38d1b1a95221fa323052edd594148c8df6ce4b13c54f16896cc9 WatchSource:0}: Error finding container 029b7dcfe19a38d1b1a95221fa323052edd594148c8df6ce4b13c54f16896cc9: Status 404 returned error can't find the container with id 029b7dcfe19a38d1b1a95221fa323052edd594148c8df6ce4b13c54f16896cc9 Nov 03 13:09:46.682781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:46.682742 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" event={"ID":"6f5d995f-6ee8-496b-80d6-1ad8c44689ec","Type":"ContainerStarted","Data":"029b7dcfe19a38d1b1a95221fa323052edd594148c8df6ce4b13c54f16896cc9"} Nov 03 13:09:46.701604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:46.701569 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:47.642809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:47.642774 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-rf94g" Nov 03 13:09:48.114940 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:48.114904 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:48.559835 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:09:48.559799 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:09:49.700982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:49.700943 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" event={"ID":"6f5d995f-6ee8-496b-80d6-1ad8c44689ec","Type":"ContainerStarted","Data":"2c5e0f9ef3c6ec41ef94de46093e5dcdd8c809f07e72af90c0c1b0067671740c"} Nov 03 13:09:49.912700 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:49.912639 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-4k9h2" podStartSLOduration=2.853453392 podStartE2EDuration="5.912622803s" totalImagesPullingTime="3.059169411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:49.865012133 +0000 UTC m=+323.059415726" watchObservedRunningTime="2026-11-03 13:09:49.912622803 +0000 UTC m=+323.107026407" Nov 03 13:09:50.720182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:50.720143 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:50.907043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:50.907002 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt"] Nov 03 13:09:50.910559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:50.910542 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" Nov 03 13:09:51.004491 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.004413 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmt7\" (UniqueName: \"kubernetes.io/projected/199f6b24-0d49-4637-b048-17b035f48627-kube-api-access-wlmt7\") pod \"dns-operator-controller-manager-8557cbdb84-fvdmt\" (UID: \"199f6b24-0d49-4637-b048-17b035f48627\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" Nov 03 13:09:51.006416 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.006390 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 13:09:51.006549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.006395 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 13:09:51.006549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.006464 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xxqx6\"" Nov 03 13:09:51.006549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.006512 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 13:09:51.008787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.008765 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt"] Nov 03 13:09:51.105688 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.105635 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmt7\" (UniqueName: \"kubernetes.io/projected/199f6b24-0d49-4637-b048-17b035f48627-kube-api-access-wlmt7\") pod \"dns-operator-controller-manager-8557cbdb84-fvdmt\" (UID: \"199f6b24-0d49-4637-b048-17b035f48627\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" Nov 03 13:09:51.166082 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.166033 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmt7\" (UniqueName: \"kubernetes.io/projected/199f6b24-0d49-4637-b048-17b035f48627-kube-api-access-wlmt7\") pod \"dns-operator-controller-manager-8557cbdb84-fvdmt\" (UID: \"199f6b24-0d49-4637-b048-17b035f48627\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" Nov 03 13:09:51.222761 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.222731 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" Nov 03 13:09:51.404914 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.404877 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:51.658796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.658768 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt"] Nov 03 13:09:51.660695 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:09:51.660667 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199f6b24_0d49_4637_b048_17b035f48627.slice/crio-fbd085e2370e68cb2cdd8a20a901f0ae1ed4929ce5b341c7758449445f424723 WatchSource:0}: Error finding container fbd085e2370e68cb2cdd8a20a901f0ae1ed4929ce5b341c7758449445f424723: Status 404 returned error can't find the container with id fbd085e2370e68cb2cdd8a20a901f0ae1ed4929ce5b341c7758449445f424723 Nov 03 13:09:51.674033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.674004 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:51.711754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.711725 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" event={"ID":"199f6b24-0d49-4637-b048-17b035f48627","Type":"ContainerStarted","Data":"fbd085e2370e68cb2cdd8a20a901f0ae1ed4929ce5b341c7758449445f424723"} Nov 03 13:09:51.839615 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:51.839583 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:53.946949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:53.946917 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:54.192570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.192532 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd"] Nov 03 13:09:54.194902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.194880 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" Nov 03 13:09:54.232029 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.231994 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv524\" (UniqueName: \"kubernetes.io/projected/1d3c8424-3ae1-488e-b37f-1d2a5d6c5199-kube-api-access-vv524\") pod \"authorino-operator-7b7bb4bc77-qkwsd\" (UID: \"1d3c8424-3ae1-488e-b37f-1d2a5d6c5199\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" Nov 03 13:09:54.293673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.293612 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f2zpg\"" Nov 03 13:09:54.296429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.296347 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd"] Nov 03 13:09:54.332853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.332819 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv524\" (UniqueName: \"kubernetes.io/projected/1d3c8424-3ae1-488e-b37f-1d2a5d6c5199-kube-api-access-vv524\") pod \"authorino-operator-7b7bb4bc77-qkwsd\" (UID: \"1d3c8424-3ae1-488e-b37f-1d2a5d6c5199\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" Nov 03 13:09:54.403843 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.403809 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv524\" (UniqueName: \"kubernetes.io/projected/1d3c8424-3ae1-488e-b37f-1d2a5d6c5199-kube-api-access-vv524\") pod \"authorino-operator-7b7bb4bc77-qkwsd\" (UID: \"1d3c8424-3ae1-488e-b37f-1d2a5d6c5199\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" Nov 03 13:09:54.487510 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.487419 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:54.510048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.510017 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" Nov 03 13:09:54.908564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.908534 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd"] Nov 03 13:09:54.909652 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:09:54.909611 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3c8424_3ae1_488e_b37f_1d2a5d6c5199.slice/crio-35ddad437a277348630b784252d7d1538749070791423c9e11e035c7d33ba515 WatchSource:0}: Error finding container 35ddad437a277348630b784252d7d1538749070791423c9e11e035c7d33ba515: Status 404 returned error can't find the container with id 35ddad437a277348630b784252d7d1538749070791423c9e11e035c7d33ba515 Nov 03 13:09:54.924675 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:54.924643 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:55.289288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:55.289235 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:55.731765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:55.731729 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" event={"ID":"1d3c8424-3ae1-488e-b37f-1d2a5d6c5199","Type":"ContainerStarted","Data":"35ddad437a277348630b784252d7d1538749070791423c9e11e035c7d33ba515"} Nov 03 13:09:56.737835 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:56.737797 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" event={"ID":"199f6b24-0d49-4637-b048-17b035f48627","Type":"ContainerStarted","Data":"e55ef6a09e46eec5f73a97b9cd61764e978e3b7e3096067440331a1386ec146a"} Nov 03 13:09:56.738270 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:56.737941 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" Nov 03 13:09:56.851375 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:56.851309 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" podStartSLOduration=2.775845138 podStartE2EDuration="6.851294956s" totalImagesPullingTime="4.075449818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:56.848967156 +0000 UTC m=+330.043370749" watchObservedRunningTime="2026-11-03 13:09:56.851294956 +0000 UTC m=+330.045698549" Nov 03 13:09:56.876258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:56.876226 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:57.462267 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:57.462221 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9"] Nov 03 13:09:57.467448 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:57.467421 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" Nov 03 13:09:57.520154 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:57.520119 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9"] Nov 03 13:09:57.564185 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:57.564149 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-llqq7\"" Nov 03 13:09:57.566518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:57.566486 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64pj\" (UniqueName: \"kubernetes.io/projected/7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a-kube-api-access-d64pj\") pod \"limitador-operator-controller-manager-5778bb4449-mvjp9\" (UID: \"7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" Nov 03 13:09:57.667040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:57.667001 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64pj\" (UniqueName: \"kubernetes.io/projected/7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a-kube-api-access-d64pj\") pod \"limitador-operator-controller-manager-5778bb4449-mvjp9\" (UID: \"7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" Nov 03 13:09:57.720775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:57.720707 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64pj\" (UniqueName: \"kubernetes.io/projected/7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a-kube-api-access-d64pj\") pod \"limitador-operator-controller-manager-5778bb4449-mvjp9\" (UID: \"7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" Nov 03 13:09:57.782508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:57.782483 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" Nov 03 13:09:58.163028 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:58.163000 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9"] Nov 03 13:09:58.164803 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:09:58.164775 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0adcf2_59cc_4afa_b3ed_b403b1d1c67a.slice/crio-c25ebc45010ba06435221bbd3f2b8d25fd292f393b8f07d6aa15bf7a4cc570d3 WatchSource:0}: Error finding container c25ebc45010ba06435221bbd3f2b8d25fd292f393b8f07d6aa15bf7a4cc570d3: Status 404 returned error can't find the container with id c25ebc45010ba06435221bbd3f2b8d25fd292f393b8f07d6aa15bf7a4cc570d3 Nov 03 13:09:58.234573 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:58.234540 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:58.704548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:58.704513 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:58.748619 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:58.748586 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" event={"ID":"7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a","Type":"ContainerStarted","Data":"c25ebc45010ba06435221bbd3f2b8d25fd292f393b8f07d6aa15bf7a4cc570d3"} Nov 03 13:09:58.750260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:58.750230 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" event={"ID":"1d3c8424-3ae1-488e-b37f-1d2a5d6c5199","Type":"ContainerStarted","Data":"381ff3f009a66333d2c4c9c3388931eb90fd50af440765962b11500870709242"} Nov 03 13:09:58.750390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:58.750367 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" Nov 03 13:09:58.829102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:58.829046 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:09:58.858967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:58.858905 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" podStartSLOduration=2.010257528 podStartE2EDuration="4.858889346s" totalImagesPullingTime="2.848631818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:58.857679389 +0000 UTC m=+332.052082984" watchObservedRunningTime="2026-11-03 13:09:58.858889346 +0000 UTC m=+332.053292940" Nov 03 13:09:59.639285 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:09:59.639254 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:00.134669 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:00.134627 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:00.286558 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:00.286516 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:01.335935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:01.335842 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:01.715874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:01.715836 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:01.766945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:01.766908 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" event={"ID":"7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a","Type":"ContainerStarted","Data":"6e6b7e0a40c6326f17dcd06269dbd27fca7dff0e79d3b1bd7ba73fa5a5da4278"} Nov 03 13:10:01.767168 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:01.767019 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" Nov 03 13:10:01.842624 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:01.842596 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:01.875836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:01.875780 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" podStartSLOduration=2.033746091 podStartE2EDuration="4.875767551s" totalImagesPullingTime="2.84202146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:01.87386292 +0000 UTC m=+335.068266514" watchObservedRunningTime="2026-11-03 13:10:01.875767551 +0000 UTC m=+335.070171144" Nov 03 13:10:03.190117 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.190065 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7"] Nov 03 13:10:03.195524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.195492 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:03.255483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.255451 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7"] Nov 03 13:10:03.316666 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.316630 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zk57w\"" Nov 03 13:10:03.321061 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.321035 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4h4x\" (UniqueName: \"kubernetes.io/projected/27a5a362-b740-42a2-8f27-c5a918dfba78-kube-api-access-b4h4x\") pod \"kuadrant-operator-controller-manager-678d87766f-8h2h7\" (UID: \"27a5a362-b740-42a2-8f27-c5a918dfba78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:03.321165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.321114 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a5a362-b740-42a2-8f27-c5a918dfba78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8h2h7\" (UID: \"27a5a362-b740-42a2-8f27-c5a918dfba78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:03.421888 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.421853 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4h4x\" (UniqueName: \"kubernetes.io/projected/27a5a362-b740-42a2-8f27-c5a918dfba78-kube-api-access-b4h4x\") pod \"kuadrant-operator-controller-manager-678d87766f-8h2h7\" (UID: \"27a5a362-b740-42a2-8f27-c5a918dfba78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:03.422029 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.421931 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a5a362-b740-42a2-8f27-c5a918dfba78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8h2h7\" (UID: \"27a5a362-b740-42a2-8f27-c5a918dfba78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:03.422301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.422282 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a5a362-b740-42a2-8f27-c5a918dfba78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8h2h7\" (UID: \"27a5a362-b740-42a2-8f27-c5a918dfba78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:03.476686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.476624 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4h4x\" (UniqueName: \"kubernetes.io/projected/27a5a362-b740-42a2-8f27-c5a918dfba78-kube-api-access-b4h4x\") pod \"kuadrant-operator-controller-manager-678d87766f-8h2h7\" (UID: \"27a5a362-b740-42a2-8f27-c5a918dfba78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:03.509450 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.509420 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:03.972101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:03.972054 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7"] Nov 03 13:10:04.004907 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:10:04.004878 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a5a362_b740_42a2_8f27_c5a918dfba78.slice/crio-28092461ec2dfd86418846b870c06386c316562b8f01a137aa61aa37806ddcda WatchSource:0}: Error finding container 28092461ec2dfd86418846b870c06386c316562b8f01a137aa61aa37806ddcda: Status 404 returned error can't find the container with id 28092461ec2dfd86418846b870c06386c316562b8f01a137aa61aa37806ddcda Nov 03 13:10:04.139289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:04.139248 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:04.274784 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:04.274716 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:04.675489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:04.675458 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:04.781736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:04.781691 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" event={"ID":"27a5a362-b740-42a2-8f27-c5a918dfba78","Type":"ContainerStarted","Data":"28092461ec2dfd86418846b870c06386c316562b8f01a137aa61aa37806ddcda"} Nov 03 13:10:04.810665 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:04.810631 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:05.537812 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:05.537777 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:05.688483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:05.688450 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:07.745278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:07.745245 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-fvdmt" Nov 03 13:10:08.477530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:08.477478 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:08.729692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:08.729606 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:08.973954 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:08.973914 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:09.456676 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:09.456639 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:09.588031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:09.588004 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:09.721896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:09.721782 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:09.758025 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:09.757997 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-qkwsd" Nov 03 13:10:09.812379 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:09.812340 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" event={"ID":"27a5a362-b740-42a2-8f27-c5a918dfba78","Type":"ContainerStarted","Data":"c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb"} Nov 03 13:10:09.812537 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:09.812399 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:09.981856 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:09.981742 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" podStartSLOduration=1.4606824600000001 podStartE2EDuration="6.981721749s" totalImagesPullingTime="5.521039289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:09.981305293 +0000 UTC m=+343.175708888" watchObservedRunningTime="2026-11-03 13:10:09.981721749 +0000 UTC m=+343.176125345" Nov 03 13:10:10.806712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:10.806667 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:10.962296 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:10.962248 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:11.105208 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.105111 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:11.363209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.363127 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl"] Nov 03 13:10:11.365926 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.365906 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.415610 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.415566 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9s5z9\"" Nov 03 13:10:11.457416 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.457371 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:11.463764 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.463732 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl"] Nov 03 13:10:11.494419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494379 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.494419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494419 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.494692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494438 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.494692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494608 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.494692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494643 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.494802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494741 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.494802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494765 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.494873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494803 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.494873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.494860 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbfq\" (UniqueName: \"kubernetes.io/projected/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-kube-api-access-dvbfq\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595617 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595669 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595700 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595717 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbfq\" (UniqueName: \"kubernetes.io/projected/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-kube-api-access-dvbfq\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595743 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595767 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595789 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595842 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.595919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.595864 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.596311 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.596262 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.596412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.596388 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.596485 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.596436 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.596624 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.596606 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.596706 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.596643 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.598078 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.598042 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.598193 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.598174 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.654851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.654783 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbfq\" (UniqueName: \"kubernetes.io/projected/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-kube-api-access-dvbfq\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.655050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.655028 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fc46679f7cttvl\" (UID: \"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.677862 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.677831 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:11.682725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:11.682689 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:12.091971 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.091945 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl"] Nov 03 13:10:12.093949 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:10:12.093923 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff10ee8_0bd0_465b_ba31_cf54dbcbd6e8.slice/crio-9302598cb8814d6355a89f60a9a48f6bbb3ca0edfc85b593caf03196c441d01a WatchSource:0}: Error finding container 9302598cb8814d6355a89f60a9a48f6bbb3ca0edfc85b593caf03196c441d01a: Status 404 returned error can't find the container with id 9302598cb8814d6355a89f60a9a48f6bbb3ca0edfc85b593caf03196c441d01a Nov 03 13:10:12.096178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.096144 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:10:12.096275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.096229 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:10:12.096335 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.096276 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:10:12.308767 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.308725 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:12.450809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.450762 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:12.774368 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.774281 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mvjp9" Nov 03 13:10:12.827917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.827878 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" event={"ID":"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8","Type":"ContainerStarted","Data":"a898a03426175fceb0b406bd1e157dfcfe16f5164441b760157682f72f55c84e"} Nov 03 13:10:12.828163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.827936 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" event={"ID":"2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8","Type":"ContainerStarted","Data":"9302598cb8814d6355a89f60a9a48f6bbb3ca0edfc85b593caf03196c441d01a"} Nov 03 13:10:12.995294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:12.995239 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" podStartSLOduration=1.995222968 podStartE2EDuration="1.995222968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:12.993859138 +0000 UTC m=+346.188262744" watchObservedRunningTime="2026-11-03 13:10:12.995222968 +0000 UTC m=+346.189626598" Nov 03 13:10:13.678542 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:13.678507 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:13.683303 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:13.683280 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:13.833224 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:13.833184 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:13.834164 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:13.834141 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fc46679f7cttvl" Nov 03 13:10:17.034543 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:17.034482 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:17.160765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:17.160730 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:17.707425 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:17.707385 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:18.534844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:18.534811 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:18.564585 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:10:18.564526 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:10:20.819748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:20.819716 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 13:10:23.658584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:23.658548 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:23.805519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:23.805486 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:26.343248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:26.343215 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:26.468391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:26.468361 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:26.584216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:26.584183 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:27.210612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:27.210572 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:27.567135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:27.567042 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:27.684900 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:27.684873 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:27.806357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:27.806320 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:28.794687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:28.794650 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h2pcl"] Nov 03 13:10:28.802358 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:28.802334 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" Nov 03 13:10:28.837209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:28.837180 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwlt\" (UniqueName: \"kubernetes.io/projected/00141676-1df7-4076-b3e0-778d48d1b9e1-kube-api-access-lbwlt\") pod \"authorino-57f7f8659f-h2pcl\" (UID: \"00141676-1df7-4076-b3e0-778d48d1b9e1\") " pod="kuadrant-system/authorino-57f7f8659f-h2pcl" Nov 03 13:10:28.848619 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:28.848590 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h2pcl"] Nov 03 13:10:28.892902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:28.892868 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-swnn6\"" Nov 03 13:10:28.936699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:28.936657 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:28.937683 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:28.937663 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwlt\" (UniqueName: \"kubernetes.io/projected/00141676-1df7-4076-b3e0-778d48d1b9e1-kube-api-access-lbwlt\") pod \"authorino-57f7f8659f-h2pcl\" (UID: \"00141676-1df7-4076-b3e0-778d48d1b9e1\") " pod="kuadrant-system/authorino-57f7f8659f-h2pcl" Nov 03 13:10:28.991501 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:28.991468 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwlt\" (UniqueName: \"kubernetes.io/projected/00141676-1df7-4076-b3e0-778d48d1b9e1-kube-api-access-lbwlt\") pod \"authorino-57f7f8659f-h2pcl\" (UID: \"00141676-1df7-4076-b3e0-778d48d1b9e1\") " pod="kuadrant-system/authorino-57f7f8659f-h2pcl" Nov 03 13:10:29.059292 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:29.059214 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:29.126983 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:29.126952 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" Nov 03 13:10:29.175167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:29.175132 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:29.463454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:29.463408 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:29.492759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:29.492734 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h2pcl"] Nov 03 13:10:29.494724 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:10:29.494695 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00141676_1df7_4076_b3e0_778d48d1b9e1.slice/crio-680c048bfde1fe89c013312fa204a5ea9dd376cb7725858659d3c5e512e3b243 WatchSource:0}: Error finding container 680c048bfde1fe89c013312fa204a5ea9dd376cb7725858659d3c5e512e3b243: Status 404 returned error can't find the container with id 680c048bfde1fe89c013312fa204a5ea9dd376cb7725858659d3c5e512e3b243 Nov 03 13:10:29.583951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:29.583915 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:29.912361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:29.912327 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" event={"ID":"00141676-1df7-4076-b3e0-778d48d1b9e1","Type":"ContainerStarted","Data":"680c048bfde1fe89c013312fa204a5ea9dd376cb7725858659d3c5e512e3b243"} Nov 03 13:10:31.344719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:31.344677 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mz862"] Nov 03 13:10:31.350394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:31.350372 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" Nov 03 13:10:31.354679 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:31.354648 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwdmk\" (UniqueName: \"kubernetes.io/projected/50b4fb28-6744-4fda-a45d-dd44d8e83913-kube-api-access-cwdmk\") pod \"authorino-56c6c9ddd5-mz862\" (UID: \"50b4fb28-6744-4fda-a45d-dd44d8e83913\") " pod="kuadrant-system/authorino-56c6c9ddd5-mz862" Nov 03 13:10:31.401946 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:31.401913 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mz862"] Nov 03 13:10:31.455296 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:31.455260 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwdmk\" (UniqueName: \"kubernetes.io/projected/50b4fb28-6744-4fda-a45d-dd44d8e83913-kube-api-access-cwdmk\") pod \"authorino-56c6c9ddd5-mz862\" (UID: \"50b4fb28-6744-4fda-a45d-dd44d8e83913\") " pod="kuadrant-system/authorino-56c6c9ddd5-mz862" Nov 03 13:10:31.508776 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:31.508741 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwdmk\" (UniqueName: \"kubernetes.io/projected/50b4fb28-6744-4fda-a45d-dd44d8e83913-kube-api-access-cwdmk\") pod \"authorino-56c6c9ddd5-mz862\" (UID: \"50b4fb28-6744-4fda-a45d-dd44d8e83913\") " pod="kuadrant-system/authorino-56c6c9ddd5-mz862" Nov 03 13:10:31.663405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:31.663375 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" Nov 03 13:10:32.025415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:32.025383 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mz862"] Nov 03 13:10:32.026945 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:10:32.026901 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50b4fb28_6744_4fda_a45d_dd44d8e83913.slice/crio-56b98a16fa2b524e49bc8ffe67bbb04f408cfd76493145d7bfcef4695e512f00 WatchSource:0}: Error finding container 56b98a16fa2b524e49bc8ffe67bbb04f408cfd76493145d7bfcef4695e512f00: Status 404 returned error can't find the container with id 56b98a16fa2b524e49bc8ffe67bbb04f408cfd76493145d7bfcef4695e512f00 Nov 03 13:10:32.854788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:32.854753 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:32.927654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:32.927618 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" event={"ID":"50b4fb28-6744-4fda-a45d-dd44d8e83913","Type":"ContainerStarted","Data":"56b98a16fa2b524e49bc8ffe67bbb04f408cfd76493145d7bfcef4695e512f00"} Nov 03 13:10:32.978920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:32.978885 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:34.940434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:34.940396 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" event={"ID":"50b4fb28-6744-4fda-a45d-dd44d8e83913","Type":"ContainerStarted","Data":"4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc"} Nov 03 13:10:35.049028 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.048972 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" podStartSLOduration=1.253081547 podStartE2EDuration="4.048957144s" totalImagesPullingTime="2.795875597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:35.047850898 +0000 UTC m=+368.242254493" watchObservedRunningTime="2026-11-03 13:10:35.048957144 +0000 UTC m=+368.243360764" Nov 03 13:10:35.075200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.075118 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h2pcl"] Nov 03 13:10:35.130640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.130610 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74"] Nov 03 13:10:35.134162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.134142 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.183235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.183199 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd95a5f-d506-4461-8baf-44d26ec69181-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.183418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.183245 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfd95a5f-d506-4461-8baf-44d26ec69181-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.183418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.183329 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lws9k\" (UniqueName: \"kubernetes.io/projected/dfd95a5f-d506-4461-8baf-44d26ec69181-kube-api-access-lws9k\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.183529 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.183449 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfd95a5f-d506-4461-8baf-44d26ec69181-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.183573 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.183558 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 13:10:35.183684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.183665 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 13:10:35.183745 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.183668 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-s5jbb\"" Nov 03 13:10:35.284050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.278941 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 13:10:35.284050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.284034 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74"] Nov 03 13:10:35.284940 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.284910 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd95a5f-d506-4461-8baf-44d26ec69181-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.285104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.284956 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfd95a5f-d506-4461-8baf-44d26ec69181-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.285104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.284989 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lws9k\" (UniqueName: \"kubernetes.io/projected/dfd95a5f-d506-4461-8baf-44d26ec69181-kube-api-access-lws9k\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.285220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.285130 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfd95a5f-d506-4461-8baf-44d26ec69181-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.285441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.285418 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfd95a5f-d506-4461-8baf-44d26ec69181-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.285506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.285486 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dfd95a5f-d506-4461-8baf-44d26ec69181-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.286004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.285985 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfd95a5f-d506-4461-8baf-44d26ec69181-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.340265 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.340198 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lws9k\" (UniqueName: \"kubernetes.io/projected/dfd95a5f-d506-4461-8baf-44d26ec69181-kube-api-access-lws9k\") pod \"kserve-module-controller-manager-7566cf5fc8-8kq74\" (UID: \"dfd95a5f-d506-4461-8baf-44d26ec69181\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.447257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.447220 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:35.859225 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.859198 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74"] Nov 03 13:10:35.860739 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:10:35.860708 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd95a5f_d506_4461_8baf_44d26ec69181.slice/crio-7ddd00c45e79e1c43428c98496124731488ccf5801d8c56c432b7d9489d6f7de WatchSource:0}: Error finding container 7ddd00c45e79e1c43428c98496124731488ccf5801d8c56c432b7d9489d6f7de: Status 404 returned error can't find the container with id 7ddd00c45e79e1c43428c98496124731488ccf5801d8c56c432b7d9489d6f7de Nov 03 13:10:35.946918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:35.946877 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" event={"ID":"dfd95a5f-d506-4461-8baf-44d26ec69181","Type":"ContainerStarted","Data":"7ddd00c45e79e1c43428c98496124731488ccf5801d8c56c432b7d9489d6f7de"} Nov 03 13:10:36.612460 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:36.612404 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:36.726464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:36.726415 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:38.964429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:38.964395 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" event={"ID":"00141676-1df7-4076-b3e0-778d48d1b9e1","Type":"ContainerStarted","Data":"6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25"} Nov 03 13:10:38.964869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:38.964435 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" podUID="00141676-1df7-4076-b3e0-778d48d1b9e1" containerName="authorino" containerID="cri-o://6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25" gracePeriod=30 Nov 03 13:10:38.965993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:38.965969 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" event={"ID":"dfd95a5f-d506-4461-8baf-44d26ec69181","Type":"ContainerStarted","Data":"e98c0f63084d06055dc72db8cd5422ebf0324ba1762665d1a0fc723c57cffb37"} Nov 03 13:10:38.966141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:38.966041 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:10:39.076334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.073314 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" podStartSLOduration=2.029616653 podStartE2EDuration="11.073294596s" totalImagesPullingTime="9.043677943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:39.068807431 +0000 UTC m=+372.263211025" watchObservedRunningTime="2026-11-03 13:10:39.073294596 +0000 UTC m=+372.267698193" Nov 03 13:10:39.179741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.179685 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" podStartSLOduration=1.201826671 podStartE2EDuration="4.179670289s" totalImagesPullingTime="2.977843618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:39.178639202 +0000 UTC m=+372.373042803" watchObservedRunningTime="2026-11-03 13:10:39.179670289 +0000 UTC m=+372.374073915" Nov 03 13:10:39.225048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.225020 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" Nov 03 13:10:39.312131 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.312066 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:39.315197 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.315175 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00141676-1df7-4076-b3e0-778d48d1b9e1-kube-api-access-lbwlt\" (UniqueName: \"kubernetes.io/projected/00141676-1df7-4076-b3e0-778d48d1b9e1-kube-api-access-lbwlt\") pod \"00141676-1df7-4076-b3e0-778d48d1b9e1\" (UID: \"00141676-1df7-4076-b3e0-778d48d1b9e1\") " Nov 03 13:10:39.317325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.317301 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00141676-1df7-4076-b3e0-778d48d1b9e1-kube-api-access-lbwlt" pod "00141676-1df7-4076-b3e0-778d48d1b9e1" (UID: "00141676-1df7-4076-b3e0-778d48d1b9e1"). InnerVolumeSpecName "kube-api-access-lbwlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:10:39.416147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.416115 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwlt\" (UniqueName: \"kubernetes.io/projected/00141676-1df7-4076-b3e0-778d48d1b9e1-kube-api-access-lbwlt\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:10:39.973112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.973054 4924 generic.go:356] "Generic (PLEG): container finished" podID="00141676-1df7-4076-b3e0-778d48d1b9e1" containerID="6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25" exitCode=0 Nov 03 13:10:39.973584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.973116 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" Nov 03 13:10:39.973584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.973140 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" event={"ID":"00141676-1df7-4076-b3e0-778d48d1b9e1","Type":"ContainerDied","Data":"6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25"} Nov 03 13:10:39.973584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.973182 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h2pcl" event={"ID":"00141676-1df7-4076-b3e0-778d48d1b9e1","Type":"ContainerDied","Data":"680c048bfde1fe89c013312fa204a5ea9dd376cb7725858659d3c5e512e3b243"} Nov 03 13:10:39.973584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.973199 4924 scope.go:122] "RemoveContainer" containerID="6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25" Nov 03 13:10:39.982115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.982097 4924 scope.go:122] "RemoveContainer" containerID="6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25" Nov 03 13:10:39.982384 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:10:39.982362 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25\": container with ID starting with 6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25 not found: ID does not exist" containerID="6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25" Nov 03 13:10:39.982449 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:39.982396 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25"} err="failed to get container status \"6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25\": rpc error: code = NotFound desc = could not find container \"6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25\": container with ID starting with 6744dd8809734841659289cb9bc023265e2dfcc11f265ba842e993c9f7922b25 not found: ID does not exist" Nov 03 13:10:40.129950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:40.129915 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h2pcl"] Nov 03 13:10:40.135857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:40.135831 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h2pcl"] Nov 03 13:10:40.949264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:40.949229 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00141676-1df7-4076-b3e0-778d48d1b9e1" path="/var/lib/kubelet/pods/00141676-1df7-4076-b3e0-778d48d1b9e1/volumes" Nov 03 13:10:48.557433 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:10:48.557393 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:10:51.337227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:51.337189 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:51.511393 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:51.511351 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:52.030356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.030323 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs"] Nov 03 13:10:52.030736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.030722 4924 cpu_manager.go:414] "containerMap: removing container" podUID="00141676-1df7-4076-b3e0-778d48d1b9e1" containerName="authorino" Nov 03 13:10:52.030780 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.030737 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00141676-1df7-4076-b3e0-778d48d1b9e1" containerName="authorino" Nov 03 13:10:52.030843 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.030833 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="00141676-1df7-4076-b3e0-778d48d1b9e1" containerName="authorino" Nov 03 13:10:52.035433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.035403 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:10:52.087522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.087488 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs"] Nov 03 13:10:52.105960 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.105926 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a2dd214-4f0c-4299-aa46-d35a67280faf-data\") pod \"postgres-7cdbf6dc44-jpdjs\" (UID: \"4a2dd214-4f0c-4299-aa46-d35a67280faf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:10:52.106115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.105965 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzshg\" (UniqueName: \"kubernetes.io/projected/4a2dd214-4f0c-4299-aa46-d35a67280faf-kube-api-access-jzshg\") pod \"postgres-7cdbf6dc44-jpdjs\" (UID: \"4a2dd214-4f0c-4299-aa46-d35a67280faf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:10:52.108432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.108411 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:10:52.131150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.131128 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:10:52.131242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.131156 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:10:52.142673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.142644 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 13:10:52.177772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.177745 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dzqfb\"" Nov 03 13:10:52.207123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.207092 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a2dd214-4f0c-4299-aa46-d35a67280faf-data\") pod \"postgres-7cdbf6dc44-jpdjs\" (UID: \"4a2dd214-4f0c-4299-aa46-d35a67280faf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:10:52.207265 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.207144 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzshg\" (UniqueName: \"kubernetes.io/projected/4a2dd214-4f0c-4299-aa46-d35a67280faf-kube-api-access-jzshg\") pod \"postgres-7cdbf6dc44-jpdjs\" (UID: \"4a2dd214-4f0c-4299-aa46-d35a67280faf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:10:52.207490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.207469 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a2dd214-4f0c-4299-aa46-d35a67280faf-data\") pod \"postgres-7cdbf6dc44-jpdjs\" (UID: \"4a2dd214-4f0c-4299-aa46-d35a67280faf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:10:52.260938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.260911 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzshg\" (UniqueName: \"kubernetes.io/projected/4a2dd214-4f0c-4299-aa46-d35a67280faf-kube-api-access-jzshg\") pod \"postgres-7cdbf6dc44-jpdjs\" (UID: \"4a2dd214-4f0c-4299-aa46-d35a67280faf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:10:52.348394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.348312 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:10:52.712166 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:52.712137 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs"] Nov 03 13:10:52.714655 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:10:52.714627 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2dd214_4f0c_4299_aa46_d35a67280faf.slice/crio-93365607d72bdd4b23dc3ce385791581d7daaa618225482a3c2fe56032f4a82a WatchSource:0}: Error finding container 93365607d72bdd4b23dc3ce385791581d7daaa618225482a3c2fe56032f4a82a: Status 404 returned error can't find the container with id 93365607d72bdd4b23dc3ce385791581d7daaa618225482a3c2fe56032f4a82a Nov 03 13:10:53.043043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:53.042956 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" event={"ID":"4a2dd214-4f0c-4299-aa46-d35a67280faf","Type":"ContainerStarted","Data":"93365607d72bdd4b23dc3ce385791581d7daaa618225482a3c2fe56032f4a82a"} Nov 03 13:10:59.974839 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:10:59.974803 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8kq74" Nov 03 13:11:00.082339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:00.082300 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" event={"ID":"4a2dd214-4f0c-4299-aa46-d35a67280faf","Type":"ContainerStarted","Data":"633a7bf6391945313c7b395f658fe7acbc3227119ca8e57ae443ec1d093dd02c"} Nov 03 13:11:00.082507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:00.082419 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:11:00.236372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:00.236275 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" podStartSLOduration=2.648161757 podStartE2EDuration="9.2362602s" totalImagesPullingTime="6.588098443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:00.234783475 +0000 UTC m=+393.429187079" watchObservedRunningTime="2026-11-03 13:11:00.2362602 +0000 UTC m=+393.430663794" Nov 03 13:11:06.116822 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:06.116784 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jpdjs" Nov 03 13:11:12.874990 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:12.874886 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:12.996122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:12.996066 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:13.117008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:13.116973 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:13.242458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:13.242412 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:13.374722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:13.374671 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:18.558428 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:11:18.558382 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:11:18.886046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:18.885955 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:20.772378 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:20.772343 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:20.897315 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:20.897281 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:23.901947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:23.901910 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:24.039993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:24.039958 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:39.617366 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:39.617325 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:39.743602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:39.743566 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:39.855140 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:39.855061 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:39.975558 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:39.975528 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:40.097163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.097133 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:40.224480 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.224448 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:40.339028 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.338946 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:40.419391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.419355 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pwnhg"] Nov 03 13:11:40.421815 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.421793 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:40.460269 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.460234 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:40.482473 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.482444 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 13:11:40.517659 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.517627 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gsv68\"" Nov 03 13:11:40.520131 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.520109 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pwnhg"] Nov 03 13:11:40.524804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.524774 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx"] Nov 03 13:11:40.527419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.527399 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:40.543059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.543035 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19a01435-9815-42ed-9e2d-324a54f125bd-cert\") pod \"kserve-controller-manager-776cc66d47-pwnhg\" (UID: \"19a01435-9815-42ed-9e2d-324a54f125bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:40.543185 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.543155 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8jpj\" (UniqueName: \"kubernetes.io/projected/19a01435-9815-42ed-9e2d-324a54f125bd-kube-api-access-n8jpj\") pod \"kserve-controller-manager-776cc66d47-pwnhg\" (UID: \"19a01435-9815-42ed-9e2d-324a54f125bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:40.576489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.576461 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 13:11:40.576702 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.576544 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-thhsc\"" Nov 03 13:11:40.578626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.578606 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:40.585908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.585884 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx"] Nov 03 13:11:40.629638 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.629568 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-p9scb"] Nov 03 13:11:40.632210 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.632191 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:40.643756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.643730 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04d8cb82-be17-4415-a003-eef06a0eac25-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6j5bx\" (UID: \"04d8cb82-be17-4415-a003-eef06a0eac25\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:40.643881 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.643763 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96fpc\" (UniqueName: \"kubernetes.io/projected/04d8cb82-be17-4415-a003-eef06a0eac25-kube-api-access-96fpc\") pod \"llmisvc-controller-manager-869d5dcc5d-6j5bx\" (UID: \"04d8cb82-be17-4415-a003-eef06a0eac25\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:40.643881 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.643795 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19a01435-9815-42ed-9e2d-324a54f125bd-cert\") pod \"kserve-controller-manager-776cc66d47-pwnhg\" (UID: \"19a01435-9815-42ed-9e2d-324a54f125bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:40.643990 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.643963 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8jpj\" (UniqueName: \"kubernetes.io/projected/19a01435-9815-42ed-9e2d-324a54f125bd-kube-api-access-n8jpj\") pod \"kserve-controller-manager-776cc66d47-pwnhg\" (UID: \"19a01435-9815-42ed-9e2d-324a54f125bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:40.646281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.646226 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19a01435-9815-42ed-9e2d-324a54f125bd-cert\") pod \"kserve-controller-manager-776cc66d47-pwnhg\" (UID: \"19a01435-9815-42ed-9e2d-324a54f125bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:40.681374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.681335 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-srtq5\"" Nov 03 13:11:40.681530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.681389 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 13:11:40.688566 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.688533 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-p9scb"] Nov 03 13:11:40.712394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.712362 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8jpj\" (UniqueName: \"kubernetes.io/projected/19a01435-9815-42ed-9e2d-324a54f125bd-kube-api-access-n8jpj\") pod \"kserve-controller-manager-776cc66d47-pwnhg\" (UID: \"19a01435-9815-42ed-9e2d-324a54f125bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:40.733184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.733159 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:40.736785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.736763 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-pghpt"] Nov 03 13:11:40.739126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.739109 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:40.744929 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.744906 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04d8cb82-be17-4415-a003-eef06a0eac25-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6j5bx\" (UID: \"04d8cb82-be17-4415-a003-eef06a0eac25\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:40.745015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.744937 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96fpc\" (UniqueName: \"kubernetes.io/projected/04d8cb82-be17-4415-a003-eef06a0eac25-kube-api-access-96fpc\") pod \"llmisvc-controller-manager-869d5dcc5d-6j5bx\" (UID: \"04d8cb82-be17-4415-a003-eef06a0eac25\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:40.745015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.744963 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb-tls-certs\") pod \"model-serving-api-75bd89fdfc-p9scb\" (UID: \"d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb\") " pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:40.745169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.745138 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64rs\" (UniqueName: \"kubernetes.io/projected/d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb-kube-api-access-j64rs\") pod \"model-serving-api-75bd89fdfc-p9scb\" (UID: \"d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb\") " pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:40.747567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.747547 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04d8cb82-be17-4415-a003-eef06a0eac25-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6j5bx\" (UID: \"04d8cb82-be17-4415-a003-eef06a0eac25\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:40.788545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.788500 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qcx97\"" Nov 03 13:11:40.788669 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.788583 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 13:11:40.788669 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.788663 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 13:11:40.795982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.795948 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-pghpt"] Nov 03 13:11:40.821206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.821179 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96fpc\" (UniqueName: \"kubernetes.io/projected/04d8cb82-be17-4415-a003-eef06a0eac25-kube-api-access-96fpc\") pod \"llmisvc-controller-manager-869d5dcc5d-6j5bx\" (UID: \"04d8cb82-be17-4415-a003-eef06a0eac25\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:40.839242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.839216 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:40.846781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.846758 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ff5388d-d285-446d-9ccc-98a7be123067-cert\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:40.846859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.846789 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbbd\" (UniqueName: \"kubernetes.io/projected/3ff5388d-d285-446d-9ccc-98a7be123067-kube-api-access-ffbbd\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:40.846859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.846820 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb-tls-certs\") pod \"model-serving-api-75bd89fdfc-p9scb\" (UID: \"d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb\") " pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:40.846945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.846930 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64rs\" (UniqueName: \"kubernetes.io/projected/d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb-kube-api-access-j64rs\") pod \"model-serving-api-75bd89fdfc-p9scb\" (UID: \"d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb\") " pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:40.846983 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.846958 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff5388d-d285-446d-9ccc-98a7be123067-metrics-certs\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:40.849206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.849173 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb-tls-certs\") pod \"model-serving-api-75bd89fdfc-p9scb\" (UID: \"d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb\") " pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:40.900595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.900568 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64rs\" (UniqueName: \"kubernetes.io/projected/d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb-kube-api-access-j64rs\") pod \"model-serving-api-75bd89fdfc-p9scb\" (UID: \"d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb\") " pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:40.944236 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.944202 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:40.947636 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.947604 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ff5388d-d285-446d-9ccc-98a7be123067-cert\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:40.947718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.947646 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbbd\" (UniqueName: \"kubernetes.io/projected/3ff5388d-d285-446d-9ccc-98a7be123067-kube-api-access-ffbbd\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:40.947718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.947696 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff5388d-d285-446d-9ccc-98a7be123067-metrics-certs\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:40.950082 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.950041 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ff5388d-d285-446d-9ccc-98a7be123067-cert\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:40.950187 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:40.950155 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ff5388d-d285-446d-9ccc-98a7be123067-metrics-certs\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:41.003010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.002981 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbbd\" (UniqueName: \"kubernetes.io/projected/3ff5388d-d285-446d-9ccc-98a7be123067-kube-api-access-ffbbd\") pod \"odh-model-controller-769875db56-pghpt\" (UID: \"3ff5388d-d285-446d-9ccc-98a7be123067\") " pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:41.069021 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.068985 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:41.097051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.097019 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pwnhg"] Nov 03 13:11:41.104058 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:11:41.104027 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a01435_9815_42ed_9e2d_324a54f125bd.slice/crio-0e2a12d65653b418232043944b125b1fde976d9f6de0f74b4b60b66a6b029349 WatchSource:0}: Error finding container 0e2a12d65653b418232043944b125b1fde976d9f6de0f74b4b60b66a6b029349: Status 404 returned error can't find the container with id 0e2a12d65653b418232043944b125b1fde976d9f6de0f74b4b60b66a6b029349 Nov 03 13:11:41.200462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.200431 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx"] Nov 03 13:11:41.201796 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:11:41.201769 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04d8cb82_be17_4415_a003_eef06a0eac25.slice/crio-39aed180b5b3c5931bec782b19e1251c9495bc143d0d7e3fc7ca6d29abbbd3ba WatchSource:0}: Error finding container 39aed180b5b3c5931bec782b19e1251c9495bc143d0d7e3fc7ca6d29abbbd3ba: Status 404 returned error can't find the container with id 39aed180b5b3c5931bec782b19e1251c9495bc143d0d7e3fc7ca6d29abbbd3ba Nov 03 13:11:41.292980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.292944 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" event={"ID":"04d8cb82-be17-4415-a003-eef06a0eac25","Type":"ContainerStarted","Data":"39aed180b5b3c5931bec782b19e1251c9495bc143d0d7e3fc7ca6d29abbbd3ba"} Nov 03 13:11:41.294111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.294054 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" event={"ID":"19a01435-9815-42ed-9e2d-324a54f125bd","Type":"ContainerStarted","Data":"0e2a12d65653b418232043944b125b1fde976d9f6de0f74b4b60b66a6b029349"} Nov 03 13:11:41.308732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.308707 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-p9scb"] Nov 03 13:11:41.309984 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:11:41.309959 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e4cc55_7a0b_4bf8_bbad_355d9f45a9bb.slice/crio-6e52a02ea6d9f94845017300b660d410097f9c333f961cd6ce83ccbb1c4408a2 WatchSource:0}: Error finding container 6e52a02ea6d9f94845017300b660d410097f9c333f961cd6ce83ccbb1c4408a2: Status 404 returned error can't find the container with id 6e52a02ea6d9f94845017300b660d410097f9c333f961cd6ce83ccbb1c4408a2 Nov 03 13:11:41.433804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.433777 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-pghpt"] Nov 03 13:11:41.436236 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:11:41.436199 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff5388d_d285_446d_9ccc_98a7be123067.slice/crio-1685aa92bc61fdd58f47714aebec6728844f6ec66ed6b7779f8ddf687c709b1f WatchSource:0}: Error finding container 1685aa92bc61fdd58f47714aebec6728844f6ec66ed6b7779f8ddf687c709b1f: Status 404 returned error can't find the container with id 1685aa92bc61fdd58f47714aebec6728844f6ec66ed6b7779f8ddf687c709b1f Nov 03 13:11:41.676690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:41.676651 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:42.305114 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:42.305055 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" event={"ID":"d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb","Type":"ContainerStarted","Data":"6e52a02ea6d9f94845017300b660d410097f9c333f961cd6ce83ccbb1c4408a2"} Nov 03 13:11:42.306798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:42.306766 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-pghpt" event={"ID":"3ff5388d-d285-446d-9ccc-98a7be123067","Type":"ContainerStarted","Data":"1685aa92bc61fdd58f47714aebec6728844f6ec66ed6b7779f8ddf687c709b1f"} Nov 03 13:11:45.329772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.329729 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" event={"ID":"d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb","Type":"ContainerStarted","Data":"dc21c2c0467ed41b39e0da8cc58457a1fbbc5644af57fb5b07d8e608118af28f"} Nov 03 13:11:45.330262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.329806 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:45.331900 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.331867 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" event={"ID":"04d8cb82-be17-4415-a003-eef06a0eac25","Type":"ContainerStarted","Data":"6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108"} Nov 03 13:11:45.332035 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.331936 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:11:45.333698 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.333673 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" event={"ID":"19a01435-9815-42ed-9e2d-324a54f125bd","Type":"ContainerStarted","Data":"762bde39110a9e8d165c89ba341917c8f07c7365255a318ffc6a4ccd7451165f"} Nov 03 13:11:45.333816 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.333725 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:11:45.439817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.439761 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" podStartSLOduration=2.118639569 podStartE2EDuration="5.439745956s" totalImagesPullingTime="3.321106387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:45.436875354 +0000 UTC m=+438.631278959" watchObservedRunningTime="2026-11-03 13:11:45.439745956 +0000 UTC m=+438.634149549" Nov 03 13:11:45.544914 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.544387 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" podStartSLOduration=2.290084234 podStartE2EDuration="5.544370244s" totalImagesPullingTime="3.25428601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:45.543845772 +0000 UTC m=+438.738249381" watchObservedRunningTime="2026-11-03 13:11:45.544370244 +0000 UTC m=+438.738773839" Nov 03 13:11:45.651909 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:45.651802 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" podStartSLOduration=2.114632543 podStartE2EDuration="5.651785312s" totalImagesPullingTime="3.537152769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:45.650638731 +0000 UTC m=+438.845042325" watchObservedRunningTime="2026-11-03 13:11:45.651785312 +0000 UTC m=+438.846188908" Nov 03 13:11:46.340427 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:46.340388 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-pghpt" event={"ID":"3ff5388d-d285-446d-9ccc-98a7be123067","Type":"ContainerStarted","Data":"5cbb7741e1d43ae27973052135ea5154a3cd2c8c5b07bbe3f780400aab1e7361"} Nov 03 13:11:46.340950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:46.340832 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:11:46.449856 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:46.449794 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-pghpt" podStartSLOduration=2.386988571 podStartE2EDuration="6.449778024s" totalImagesPullingTime="4.062789453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:46.447195692 +0000 UTC m=+439.641599287" watchObservedRunningTime="2026-11-03 13:11:46.449778024 +0000 UTC m=+439.644181617" Nov 03 13:11:46.701524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:46.701483 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:46.902747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:46.902714 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:47.023087 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:47.022985 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:47.150497 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:47.150448 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:47.269378 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:47.269343 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:47.383155 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:47.383080 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:47.496293 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:47.496262 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:47.615477 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:47.615407 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:47.737609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:47.737571 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:47.850733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:47.850698 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:48.558575 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:11:48.558532 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:11:49.146665 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:49.146628 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:49.261985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:49.261942 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:51.275902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:51.275864 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:51.440614 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:51.440577 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:51.554383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:51.554279 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:51.668319 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:51.668281 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:11:56.344593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:56.344564 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-p9scb" Nov 03 13:11:58.358415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:11:58.358379 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-pghpt" Nov 03 13:12:06.048954 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.048911 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:12:06.177935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.177904 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:12:06.240451 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.240420 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 03 13:12:06.240580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.240481 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 03 13:12:06.445779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.445741 4924 generic.go:356] "Generic (PLEG): container finished" podID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerID="61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea" exitCode=0 Nov 03 13:12:06.445940 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.445816 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" event={"ID":"cbfb959d-44dd-4c99-8b90-888ae2d2305e","Type":"ContainerDied","Data":"61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea"} Nov 03 13:12:06.446214 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.446189 4924 scope.go:122] "RemoveContainer" containerID="61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea" Nov 03 13:12:06.447258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.447229 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:12:06.447331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.447297 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:12:06.447331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:06.447327 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:12:07.456572 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:07.456535 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" event={"ID":"cbfb959d-44dd-4c99-8b90-888ae2d2305e","Type":"ContainerStarted","Data":"56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1"} Nov 03 13:12:07.780958 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:07.780860 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:12:07.781161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:07.781115 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 03 13:12:07.781241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:07.781183 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 03 13:12:08.781446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:08.781410 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 03 13:12:08.781865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:08.781464 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 03 13:12:09.781291 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:09.781255 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 03 13:12:09.781466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:09.781306 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 03 13:12:10.781283 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:10.781242 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 03 13:12:10.781461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:10.781307 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 03 13:12:11.781508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:11.781468 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Nov 03 13:12:11.781917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:11.781525 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Nov 03 13:12:12.785366 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:12.785333 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:12:12.785860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:12.785730 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:12:12.786543 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:12.786520 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:12:12.827844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:12.827811 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mz862"] Nov 03 13:12:12.828110 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:12.828054 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" podUID="50b4fb28-6744-4fda-a45d-dd44d8e83913" containerName="authorino" containerID="cri-o://4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc" gracePeriod=30 Nov 03 13:12:13.073511 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.073486 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" Nov 03 13:12:13.100032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.100003 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b4fb28-6744-4fda-a45d-dd44d8e83913-kube-api-access-cwdmk\" (UniqueName: \"kubernetes.io/projected/50b4fb28-6744-4fda-a45d-dd44d8e83913-kube-api-access-cwdmk\") pod \"50b4fb28-6744-4fda-a45d-dd44d8e83913\" (UID: \"50b4fb28-6744-4fda-a45d-dd44d8e83913\") " Nov 03 13:12:13.101992 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.101968 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b4fb28-6744-4fda-a45d-dd44d8e83913-kube-api-access-cwdmk" pod "50b4fb28-6744-4fda-a45d-dd44d8e83913" (UID: "50b4fb28-6744-4fda-a45d-dd44d8e83913"). InnerVolumeSpecName "kube-api-access-cwdmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:12:13.201154 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.201125 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwdmk\" (UniqueName: \"kubernetes.io/projected/50b4fb28-6744-4fda-a45d-dd44d8e83913-kube-api-access-cwdmk\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:12:13.490385 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.490343 4924 generic.go:356] "Generic (PLEG): container finished" podID="50b4fb28-6744-4fda-a45d-dd44d8e83913" containerID="4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc" exitCode=0 Nov 03 13:12:13.490385 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.490385 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" Nov 03 13:12:13.490640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.490436 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" event={"ID":"50b4fb28-6744-4fda-a45d-dd44d8e83913","Type":"ContainerDied","Data":"4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc"} Nov 03 13:12:13.490640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.490475 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mz862" event={"ID":"50b4fb28-6744-4fda-a45d-dd44d8e83913","Type":"ContainerDied","Data":"56b98a16fa2b524e49bc8ffe67bbb04f408cfd76493145d7bfcef4695e512f00"} Nov 03 13:12:13.490640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.490491 4924 scope.go:122] "RemoveContainer" containerID="4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc" Nov 03 13:12:13.499753 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.499736 4924 scope.go:122] "RemoveContainer" containerID="4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc" Nov 03 13:12:13.500019 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:12:13.500000 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc\": container with ID starting with 4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc not found: ID does not exist" containerID="4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc" Nov 03 13:12:13.500066 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.500028 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc"} err="failed to get container status \"4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc\": rpc error: code = NotFound desc = could not find container \"4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc\": container with ID starting with 4d409a0712847f4f6fd15e27d87b239ba414853f16ea0ce3fe6450668d276abc not found: ID does not exist" Nov 03 13:12:13.642721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.642685 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mz862"] Nov 03 13:12:13.646335 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:13.646303 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mz862"] Nov 03 13:12:14.949910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:14.949873 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b4fb28-6744-4fda-a45d-dd44d8e83913" path="/var/lib/kubelet/pods/50b4fb28-6744-4fda-a45d-dd44d8e83913/volumes" Nov 03 13:12:16.343083 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:16.343030 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:12:16.346273 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:16.346250 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pwnhg" Nov 03 13:12:18.558358 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:12:18.558325 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:12:19.227138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:19.227100 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:12:25.252485 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:25.252451 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:12:48.557956 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:12:48.557878 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:12:54.072060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.072025 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m"] Nov 03 13:12:54.072469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.072451 4924 cpu_manager.go:414] "containerMap: removing container" podUID="50b4fb28-6744-4fda-a45d-dd44d8e83913" containerName="authorino" Nov 03 13:12:54.072469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.072464 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b4fb28-6744-4fda-a45d-dd44d8e83913" containerName="authorino" Nov 03 13:12:54.072554 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.072544 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b4fb28-6744-4fda-a45d-dd44d8e83913" containerName="authorino" Nov 03 13:12:54.074685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.074667 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:54.129338 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.129304 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m"] Nov 03 13:12:54.273505 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.273471 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde28c41-ee98-42d6-ad61-6d08466d2fc9-cert\") pod \"llmisvc-controller-manager-6466c554d8-xbk2m\" (UID: \"bde28c41-ee98-42d6-ad61-6d08466d2fc9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:54.273702 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.273539 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4b4n\" (UniqueName: \"kubernetes.io/projected/bde28c41-ee98-42d6-ad61-6d08466d2fc9-kube-api-access-p4b4n\") pod \"llmisvc-controller-manager-6466c554d8-xbk2m\" (UID: \"bde28c41-ee98-42d6-ad61-6d08466d2fc9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:54.374263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.374178 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4b4n\" (UniqueName: \"kubernetes.io/projected/bde28c41-ee98-42d6-ad61-6d08466d2fc9-kube-api-access-p4b4n\") pod \"llmisvc-controller-manager-6466c554d8-xbk2m\" (UID: \"bde28c41-ee98-42d6-ad61-6d08466d2fc9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:54.374419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.374313 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde28c41-ee98-42d6-ad61-6d08466d2fc9-cert\") pod \"llmisvc-controller-manager-6466c554d8-xbk2m\" (UID: \"bde28c41-ee98-42d6-ad61-6d08466d2fc9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:54.376728 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.376705 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde28c41-ee98-42d6-ad61-6d08466d2fc9-cert\") pod \"llmisvc-controller-manager-6466c554d8-xbk2m\" (UID: \"bde28c41-ee98-42d6-ad61-6d08466d2fc9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:54.428301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.428266 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4b4n\" (UniqueName: \"kubernetes.io/projected/bde28c41-ee98-42d6-ad61-6d08466d2fc9-kube-api-access-p4b4n\") pod \"llmisvc-controller-manager-6466c554d8-xbk2m\" (UID: \"bde28c41-ee98-42d6-ad61-6d08466d2fc9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:54.686127 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:54.686090 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:55.048866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:55.048839 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m"] Nov 03 13:12:55.050375 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:12:55.050349 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde28c41_ee98_42d6_ad61_6d08466d2fc9.slice/crio-5fdd918e13172cb89fb12d16d26612e36f970c8b700d011c0dbd7c73a9b6bf64 WatchSource:0}: Error finding container 5fdd918e13172cb89fb12d16d26612e36f970c8b700d011c0dbd7c73a9b6bf64: Status 404 returned error can't find the container with id 5fdd918e13172cb89fb12d16d26612e36f970c8b700d011c0dbd7c73a9b6bf64 Nov 03 13:12:55.713032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:55.712996 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" event={"ID":"bde28c41-ee98-42d6-ad61-6d08466d2fc9","Type":"ContainerStarted","Data":"5fdd918e13172cb89fb12d16d26612e36f970c8b700d011c0dbd7c73a9b6bf64"} Nov 03 13:12:56.719720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:56.719685 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" event={"ID":"bde28c41-ee98-42d6-ad61-6d08466d2fc9","Type":"ContainerStarted","Data":"d677a69c10d53c198ea284a096fb80ba8f3c1a591ad98db4bd12620d0f7fe9d6"} Nov 03 13:12:56.720135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:56.719748 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:12:56.826763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:12:56.826707 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" podStartSLOduration=2.061217746 podStartE2EDuration="2.826690716s" totalImagesPullingTime="765.47297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:12:56.825797556 +0000 UTC m=+510.020201150" watchObservedRunningTime="2026-11-03 13:12:56.826690716 +0000 UTC m=+510.021094311" Nov 03 13:13:18.557649 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:13:18.557611 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:13:19.559750 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:19.559672 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:27.727484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:27.727453 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xbk2m" Nov 03 13:13:27.883519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:27.883486 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx"] Nov 03 13:13:27.883760 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:27.883736 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" podUID="04d8cb82-be17-4415-a003-eef06a0eac25" containerName="manager" containerID="cri-o://6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108" gracePeriod=30 Nov 03 13:13:28.133603 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.133577 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:13:28.247779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.247749 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d8cb82-be17-4415-a003-eef06a0eac25-cert\" (UniqueName: \"kubernetes.io/secret/04d8cb82-be17-4415-a003-eef06a0eac25-cert\") pod \"04d8cb82-be17-4415-a003-eef06a0eac25\" (UID: \"04d8cb82-be17-4415-a003-eef06a0eac25\") " Nov 03 13:13:28.247971 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.247793 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d8cb82-be17-4415-a003-eef06a0eac25-kube-api-access-96fpc\" (UniqueName: \"kubernetes.io/projected/04d8cb82-be17-4415-a003-eef06a0eac25-kube-api-access-96fpc\") pod \"04d8cb82-be17-4415-a003-eef06a0eac25\" (UID: \"04d8cb82-be17-4415-a003-eef06a0eac25\") " Nov 03 13:13:28.250200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.250170 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d8cb82-be17-4415-a003-eef06a0eac25-cert" pod "04d8cb82-be17-4415-a003-eef06a0eac25" (UID: "04d8cb82-be17-4415-a003-eef06a0eac25"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:13:28.250498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.250482 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d8cb82-be17-4415-a003-eef06a0eac25-kube-api-access-96fpc" pod "04d8cb82-be17-4415-a003-eef06a0eac25" (UID: "04d8cb82-be17-4415-a003-eef06a0eac25"). InnerVolumeSpecName "kube-api-access-96fpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:13:28.349037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.349003 4924 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04d8cb82-be17-4415-a003-eef06a0eac25-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:13:28.349037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.349034 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96fpc\" (UniqueName: \"kubernetes.io/projected/04d8cb82-be17-4415-a003-eef06a0eac25-kube-api-access-96fpc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:13:28.888801 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.888766 4924 generic.go:356] "Generic (PLEG): container finished" podID="04d8cb82-be17-4415-a003-eef06a0eac25" containerID="6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108" exitCode=0 Nov 03 13:13:28.889330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.888815 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" Nov 03 13:13:28.889330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.888859 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" event={"ID":"04d8cb82-be17-4415-a003-eef06a0eac25","Type":"ContainerDied","Data":"6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108"} Nov 03 13:13:28.889330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.888920 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx" event={"ID":"04d8cb82-be17-4415-a003-eef06a0eac25","Type":"ContainerDied","Data":"39aed180b5b3c5931bec782b19e1251c9495bc143d0d7e3fc7ca6d29abbbd3ba"} Nov 03 13:13:28.889330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.888953 4924 scope.go:122] "RemoveContainer" containerID="6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108" Nov 03 13:13:28.899400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.899383 4924 scope.go:122] "RemoveContainer" containerID="6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108" Nov 03 13:13:28.899720 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:13:28.899699 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108\": container with ID starting with 6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108 not found: ID does not exist" containerID="6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108" Nov 03 13:13:28.899781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.899733 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108"} err="failed to get container status \"6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108\": rpc error: code = NotFound desc = could not find container \"6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108\": container with ID starting with 6235c1778e649af1faf8ea962938b6961ec3976bca7236e2329ac3ce9fd33108 not found: ID does not exist" Nov 03 13:13:28.924302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:28.924273 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:29.042298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:29.042255 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx"] Nov 03 13:13:29.089364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:29.089316 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6j5bx"] Nov 03 13:13:29.227042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:29.226999 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:30.950522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:30.950487 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d8cb82-be17-4415-a003-eef06a0eac25" path="/var/lib/kubelet/pods/04d8cb82-be17-4415-a003-eef06a0eac25/volumes" Nov 03 13:13:35.359418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:35.359386 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:35.528148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:35.528113 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:48.558756 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:13:48.558718 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:13:51.596306 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:51.596258 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:52.346065 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:52.346028 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:54.539342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:54.539303 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:57.751588 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:57.751553 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:58.282397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:58.282361 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:59.156734 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:59.156702 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:59.289539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:59.289503 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:59.599342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:59.599270 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:13:59.781289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:13:59.781255 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:07.898119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:07.898015 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:15.534477 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:15.534440 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:18.564185 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:14:18.564145 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:14:19.903574 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:19.903537 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:20.167005 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:20.166969 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:21.319042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:21.319011 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:25.054684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:25.054648 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:25.560937 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:25.560906 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:27.011680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:27.011651 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:14:27.012534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:27.012516 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:14:30.935494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:30.935445 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:34.178683 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:34.178641 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:37.108667 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:37.108635 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:40.505817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:40.505781 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:14:48.560062 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:14:48.560023 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:14:56.252784 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.252739 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74"] Nov 03 13:14:56.253435 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.253372 4924 cpu_manager.go:414] "containerMap: removing container" podUID="04d8cb82-be17-4415-a003-eef06a0eac25" containerName="manager" Nov 03 13:14:56.253435 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.253393 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d8cb82-be17-4415-a003-eef06a0eac25" containerName="manager" Nov 03 13:14:56.253559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.253525 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d8cb82-be17-4415-a003-eef06a0eac25" containerName="manager" Nov 03 13:14:56.256859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.256838 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.365366 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.365334 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74"] Nov 03 13:14:56.434608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434573 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.434608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434612 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.434807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434629 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqd7m\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-kube-api-access-wqd7m\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.434807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434739 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.434807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434792 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.434908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434840 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.434908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434872 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.434908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434895 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.435004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.434913 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536151 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536190 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536225 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536256 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536285 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536360 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536390 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536413 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqd7m\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-kube-api-access-wqd7m\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536448 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536810 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536703 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536810 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536714 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536915 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536902 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.536974 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.536904 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.537415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.537385 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.538807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.538787 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.539045 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.539025 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.590181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.590151 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqd7m\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-kube-api-access-wqd7m\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.590338 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.590317 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-54t74\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:56.870052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:56.869962 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:57.283122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:57.283095 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74"] Nov 03 13:14:57.284795 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:14:57.284767 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a879b5_cce0_4eba_862b_dfbd25bb7d12.slice/crio-3d0a3637f0a36505c15fd77c61a358f5e29c3d19d1635e4cf5842c10ea020f75 WatchSource:0}: Error finding container 3d0a3637f0a36505c15fd77c61a358f5e29c3d19d1635e4cf5842c10ea020f75: Status 404 returned error can't find the container with id 3d0a3637f0a36505c15fd77c61a358f5e29c3d19d1635e4cf5842c10ea020f75 Nov 03 13:14:57.286880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:57.286846 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:57.286951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:57.286910 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:57.286951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:57.286939 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:57.375596 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:57.375556 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" event={"ID":"a9a879b5-cce0-4eba-862b-dfbd25bb7d12","Type":"ContainerStarted","Data":"c0c1b6ba95de41512352612a9d33983b9d33f44ae453c79e14caf39fc21367fa"} Nov 03 13:14:57.375770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:57.375623 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" event={"ID":"a9a879b5-cce0-4eba-862b-dfbd25bb7d12","Type":"ContainerStarted","Data":"3d0a3637f0a36505c15fd77c61a358f5e29c3d19d1635e4cf5842c10ea020f75"} Nov 03 13:14:57.490871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:57.490813 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podStartSLOduration=1.49079748 podStartE2EDuration="1.49079748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:57.488449527 +0000 UTC m=+630.682853122" watchObservedRunningTime="2026-11-03 13:14:57.49079748 +0000 UTC m=+630.685201073" Nov 03 13:14:57.870252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:57.870185 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:14:58.871733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:58.871697 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 13:14:58.872163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:58.871749 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 13:14:59.871012 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:59.870973 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 13:14:59.871218 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:14:59.871029 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 13:15:00.870955 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:00.870918 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 13:15:00.871373 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:00.870979 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 13:15:01.870837 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:01.870797 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 13:15:01.871024 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:01.870857 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 13:15:02.870734 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:02.870696 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 13:15:02.870919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:02.870751 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 13:15:03.871331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:03.871294 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 13:15:03.871714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:03.871346 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 13:15:04.871004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:04.870973 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 13:15:04.871187 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:04.871026 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 13:15:05.874235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:05.874202 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:15:05.874696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:05.874529 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:15:05.875163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:05.875146 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:15:08.339686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:08.339641 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:08.471228 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:08.471182 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:09.892028 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:09.891981 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:10.016112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:10.016053 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:18.558204 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:15:18.558168 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:15:20.248310 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:20.248266 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:39.571064 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:39.570987 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:39.695080 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:39.695034 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:39.838978 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:39.838897 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:40.114886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:40.114795 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:40.375613 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:40.375515 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:40.498476 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:40.498447 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:40.646950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:40.646871 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:40.843289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:40.843248 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:40.974575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:40.974543 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:41.412177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:41.412131 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:41.612716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:41.612679 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:41.726212 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:41.726134 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:43.178302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:43.178270 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:43.498393 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:43.498307 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:43.629997 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:43.629959 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:43.802385 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:43.802299 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:43.925142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:43.925113 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:44.411248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:44.411213 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:44.621085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:44.621041 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:44.754112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:44.754019 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:46.231378 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:46.231339 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:46.390478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:46.390443 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:46.580053 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:46.579963 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:46.923657 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:46.923627 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:47.111685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:47.111651 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:47.252022 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:47.251931 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:47.445592 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:47.445550 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:48.418814 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:48.418783 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:48.559787 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:15:48.559753 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:15:48.603650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:48.603621 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:48.734312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:48.734234 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:49.159694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:49.159662 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:49.288325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:49.288296 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:50.518998 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:50.518969 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:50.695038 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:50.695002 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:50.944648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:50.944550 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:51.806764 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:51.806729 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:51.995102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:51.995051 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:54.083438 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:54.083409 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:54.221373 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:54.221337 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:54.340987 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:54.340894 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:54.516671 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:54.516639 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:56.108396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:56.108363 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:56.264267 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:56.264232 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:56.375966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:56.375880 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:56.895259 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:56.895227 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:57.066165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.066130 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:15:57.323328 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.323289 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz"] Nov 03 13:15:57.327138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.327104 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:15:57.380295 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.380266 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz"] Nov 03 13:15:57.389913 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.389878 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e047289-890e-440c-8a63-e4561c401e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz\" (UID: \"4e047289-890e-440c-8a63-e4561c401e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:15:57.390155 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.389935 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfbvj\" (UniqueName: \"kubernetes.io/projected/4e047289-890e-440c-8a63-e4561c401e9b-kube-api-access-mfbvj\") pod \"payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz\" (UID: \"4e047289-890e-440c-8a63-e4561c401e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:15:57.423682 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.423638 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cb1bd0b2\"" Nov 03 13:15:57.423870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.423662 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cb1bd0b2-dockercfg-srwg7\"" Nov 03 13:15:57.491146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.491110 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e047289-890e-440c-8a63-e4561c401e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz\" (UID: \"4e047289-890e-440c-8a63-e4561c401e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:15:57.491313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.491175 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfbvj\" (UniqueName: \"kubernetes.io/projected/4e047289-890e-440c-8a63-e4561c401e9b-kube-api-access-mfbvj\") pod \"payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz\" (UID: \"4e047289-890e-440c-8a63-e4561c401e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:15:57.491851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.491831 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e047289-890e-440c-8a63-e4561c401e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz\" (UID: \"4e047289-890e-440c-8a63-e4561c401e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:15:57.544767 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.544731 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfbvj\" (UniqueName: \"kubernetes.io/projected/4e047289-890e-440c-8a63-e4561c401e9b-kube-api-access-mfbvj\") pod \"payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz\" (UID: \"4e047289-890e-440c-8a63-e4561c401e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:15:57.640685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:57.640590 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:15:58.007324 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:58.007298 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz"] Nov 03 13:15:58.008834 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:15:58.008796 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e047289_890e_440c_8a63_e4561c401e9b.slice/crio-2f63874297a436083fc1e293b79e47a2b7e7591207157a3e3cda4a56ef18e3d1 WatchSource:0}: Error finding container 2f63874297a436083fc1e293b79e47a2b7e7591207157a3e3cda4a56ef18e3d1: Status 404 returned error can't find the container with id 2f63874297a436083fc1e293b79e47a2b7e7591207157a3e3cda4a56ef18e3d1 Nov 03 13:15:58.010865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:58.010851 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:15:58.719711 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:15:58.719665 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" event={"ID":"4e047289-890e-440c-8a63-e4561c401e9b","Type":"ContainerStarted","Data":"2f63874297a436083fc1e293b79e47a2b7e7591207157a3e3cda4a56ef18e3d1"} Nov 03 13:16:01.438512 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:01.438474 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:01.610978 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:01.610936 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:01.723886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:01.723789 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:01.739507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:01.739459 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" event={"ID":"4e047289-890e-440c-8a63-e4561c401e9b","Type":"ContainerStarted","Data":"cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca"} Nov 03 13:16:01.739664 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:01.739616 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:16:01.847292 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:01.847234 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" podStartSLOduration=2.135323397 podStartE2EDuration="4.847220817s" totalImagesPullingTime="2.71189742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:01.845585958 +0000 UTC m=+695.039989551" watchObservedRunningTime="2026-11-03 13:16:01.847220817 +0000 UTC m=+695.041624411" Nov 03 13:16:06.401910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:06.401879 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:06.517943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:06.517873 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:06.636870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:06.636835 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:06.965220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:06.965185 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp"] Nov 03 13:16:06.968969 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:06.968951 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.065809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.065770 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a7a9081d\"" Nov 03 13:16:07.065984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.065879 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a7a9081d-dockercfg-lqf76\"" Nov 03 13:16:07.069234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.069206 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp"] Nov 03 13:16:07.086097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.086052 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjgr\" (UniqueName: \"kubernetes.io/projected/80f50249-4e42-4751-8333-8d37fd843b7a-kube-api-access-9hjgr\") pod \"payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp\" (UID: \"80f50249-4e42-4751-8333-8d37fd843b7a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.086261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.086131 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50249-4e42-4751-8333-8d37fd843b7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp\" (UID: \"80f50249-4e42-4751-8333-8d37fd843b7a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.187567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.187529 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50249-4e42-4751-8333-8d37fd843b7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp\" (UID: \"80f50249-4e42-4751-8333-8d37fd843b7a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.187934 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.187895 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjgr\" (UniqueName: \"kubernetes.io/projected/80f50249-4e42-4751-8333-8d37fd843b7a-kube-api-access-9hjgr\") pod \"payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp\" (UID: \"80f50249-4e42-4751-8333-8d37fd843b7a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.188396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.188375 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50249-4e42-4751-8333-8d37fd843b7a-plugins-config-volume\") pod \"payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp\" (UID: \"80f50249-4e42-4751-8333-8d37fd843b7a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.242892 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.242810 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjgr\" (UniqueName: \"kubernetes.io/projected/80f50249-4e42-4751-8333-8d37fd843b7a-kube-api-access-9hjgr\") pod \"payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp\" (UID: \"80f50249-4e42-4751-8333-8d37fd843b7a\") " pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.283632 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.283594 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.664371 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.664344 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp"] Nov 03 13:16:07.667134 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:16:07.667102 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f50249_4e42_4751_8333_8d37fd843b7a.slice/crio-a0579d352aa26e4ef073b4ba9f3d45238edb2ae59710cedcfe5c16aede847e48 WatchSource:0}: Error finding container a0579d352aa26e4ef073b4ba9f3d45238edb2ae59710cedcfe5c16aede847e48: Status 404 returned error can't find the container with id a0579d352aa26e4ef073b4ba9f3d45238edb2ae59710cedcfe5c16aede847e48 Nov 03 13:16:07.779025 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.778987 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" event={"ID":"80f50249-4e42-4751-8333-8d37fd843b7a","Type":"ContainerStarted","Data":"7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067"} Nov 03 13:16:07.779212 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.779039 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" event={"ID":"80f50249-4e42-4751-8333-8d37fd843b7a","Type":"ContainerStarted","Data":"a0579d352aa26e4ef073b4ba9f3d45238edb2ae59710cedcfe5c16aede847e48"} Nov 03 13:16:07.779289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.779210 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:07.886513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:07.886456 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" podStartSLOduration=1.886441237 podStartE2EDuration="1.886441237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:07.88545507 +0000 UTC m=+701.079858665" watchObservedRunningTime="2026-11-03 13:16:07.886441237 +0000 UTC m=+701.080844831" Nov 03 13:16:11.624157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:11.624125 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:11.788649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:11.788611 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:11.911241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:11.911205 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:12.753670 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:12.753640 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:16:16.967522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:16.967488 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:18.558218 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:16:18.558184 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:16:18.787010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:18.786981 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:16:20.560266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:20.560228 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:21.816825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.816785 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt"] Nov 03 13:16:21.820806 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.820783 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:21.918487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.918442 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-11ca03ad-serving-cert\"" Nov 03 13:16:21.918671 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.918526 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:16:21.918671 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.918629 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgx9m\"" Nov 03 13:16:21.919775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.919745 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:21.919775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.919775 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e42bf-8406-4111-988b-2a898926cc1a-openshift-service-ca\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:21.919991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.919968 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzzb7\" (UniqueName: \"kubernetes.io/projected/f46e42bf-8406-4111-988b-2a898926cc1a-kube-api-access-dzzb7\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:21.920037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.920011 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-metrics-certs\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:21.920943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.920924 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt"] Nov 03 13:16:21.998369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:21.998336 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:22.021561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.021516 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzb7\" (UniqueName: \"kubernetes.io/projected/f46e42bf-8406-4111-988b-2a898926cc1a-kube-api-access-dzzb7\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.021705 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.021568 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-metrics-certs\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.021705 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.021617 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.021705 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.021641 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e42bf-8406-4111-988b-2a898926cc1a-openshift-service-ca\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.021853 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:16:22.021730 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-serving-cert: secret "maas-api-e2e-ait-11ca03ad-serving-cert" not found Nov 03 13:16:22.021853 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:16:22.021813 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls podName:f46e42bf-8406-4111-988b-2a898926cc1a nodeName:}" failed. No retries permitted until 2026-11-03 13:16:22.521795547 +0000 UTC m=+715.716199125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls") pod "maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" (UID: "f46e42bf-8406-4111-988b-2a898926cc1a") : secret "maas-api-e2e-ait-11ca03ad-serving-cert" not found Nov 03 13:16:22.022534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.022510 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e42bf-8406-4111-988b-2a898926cc1a-openshift-service-ca\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.024260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.024240 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-metrics-certs\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.076787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.076714 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzb7\" (UniqueName: \"kubernetes.io/projected/f46e42bf-8406-4111-988b-2a898926cc1a-kube-api-access-dzzb7\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.174217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.174183 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:22.297509 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.297477 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:22.526999 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.526963 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.529411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.529385 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls\") pod \"maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:22.734690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:22.734653 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:23.103903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:23.103875 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt"] Nov 03 13:16:23.106108 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:16:23.106058 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46e42bf_8406_4111_988b_2a898926cc1a.slice/crio-863b68561516dc917ccfc583f8fd7308028f322f22d49056576f94b53f69d2c1 WatchSource:0}: Error finding container 863b68561516dc917ccfc583f8fd7308028f322f22d49056576f94b53f69d2c1: Status 404 returned error can't find the container with id 863b68561516dc917ccfc583f8fd7308028f322f22d49056576f94b53f69d2c1 Nov 03 13:16:23.873230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:23.873191 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" event={"ID":"f46e42bf-8406-4111-988b-2a898926cc1a","Type":"ContainerStarted","Data":"863b68561516dc917ccfc583f8fd7308028f322f22d49056576f94b53f69d2c1"} Nov 03 13:16:26.891753 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:26.891706 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" event={"ID":"f46e42bf-8406-4111-988b-2a898926cc1a","Type":"ContainerStarted","Data":"1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23"} Nov 03 13:16:26.892159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:26.891890 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:27.003025 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:27.002973 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" podStartSLOduration=2.960685209 podStartE2EDuration="6.002955786s" totalImagesPullingTime="3.042270577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:27.002554731 +0000 UTC m=+720.196958324" watchObservedRunningTime="2026-11-03 13:16:27.002955786 +0000 UTC m=+720.197359381" Nov 03 13:16:27.771725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:27.771689 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:28.777540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:28.777507 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:32.902582 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:32.902553 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:16:33.187467 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:33.187434 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:37.564787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:37.564751 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:37.680087 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:37.680038 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:38.485954 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:38.485908 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:44.095864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:44.095829 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:48.559168 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:16:48.559130 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:16:48.814906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:48.814836 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:48.934988 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:48.934957 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:49.059231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:49.059198 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:50.660289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:50.660251 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:50.924736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:50.924652 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:51.044160 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:51.044126 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:51.377907 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:51.377825 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:52.177297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:52.177256 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:52.351310 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:52.351268 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz"] Nov 03 13:16:52.612186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:52.612093 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:52.749171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:52.749142 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:52.881625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:52.881539 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:53.007359 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:53.007326 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:55.793194 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:55.793164 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:57.135208 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.135174 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:57.352605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.352556 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" podUID="4e047289-890e-440c-8a63-e4561c401e9b" containerName="payload-pre-processing" containerID="cri-o://cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca" gracePeriod=25 Nov 03 13:16:57.463475 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.463441 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:57.608623 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.608599 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:16:57.760900 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.760819 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e047289-890e-440c-8a63-e4561c401e9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e047289-890e-440c-8a63-e4561c401e9b-plugins-config-volume\") pod \"4e047289-890e-440c-8a63-e4561c401e9b\" (UID: \"4e047289-890e-440c-8a63-e4561c401e9b\") " Nov 03 13:16:57.761080 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.760914 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e047289-890e-440c-8a63-e4561c401e9b-kube-api-access-mfbvj\" (UniqueName: \"kubernetes.io/projected/4e047289-890e-440c-8a63-e4561c401e9b-kube-api-access-mfbvj\") pod \"4e047289-890e-440c-8a63-e4561c401e9b\" (UID: \"4e047289-890e-440c-8a63-e4561c401e9b\") " Nov 03 13:16:57.761352 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.761319 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e047289-890e-440c-8a63-e4561c401e9b-plugins-config-volume" pod "4e047289-890e-440c-8a63-e4561c401e9b" (UID: "4e047289-890e-440c-8a63-e4561c401e9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:57.763236 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.763213 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e047289-890e-440c-8a63-e4561c401e9b-kube-api-access-mfbvj" pod "4e047289-890e-440c-8a63-e4561c401e9b" (UID: "4e047289-890e-440c-8a63-e4561c401e9b"). InnerVolumeSpecName "kube-api-access-mfbvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:57.793195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.793167 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:57.861851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.861818 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfbvj\" (UniqueName: \"kubernetes.io/projected/4e047289-890e-440c-8a63-e4561c401e9b-kube-api-access-mfbvj\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:57.861851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:57.861846 4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e047289-890e-440c-8a63-e4561c401e9b-plugins-config-volume\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:58.060454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.060368 4924 generic.go:356] "Generic (PLEG): container finished" podID="4e047289-890e-440c-8a63-e4561c401e9b" containerID="cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca" exitCode=0 Nov 03 13:16:58.060454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.060425 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" Nov 03 13:16:58.060678 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.060460 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" event={"ID":"4e047289-890e-440c-8a63-e4561c401e9b","Type":"ContainerDied","Data":"cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca"} Nov 03 13:16:58.060678 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.060523 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz" event={"ID":"4e047289-890e-440c-8a63-e4561c401e9b","Type":"ContainerDied","Data":"2f63874297a436083fc1e293b79e47a2b7e7591207157a3e3cda4a56ef18e3d1"} Nov 03 13:16:58.060678 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.060552 4924 scope.go:122] "RemoveContainer" containerID="cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca" Nov 03 13:16:58.069902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.069886 4924 scope.go:122] "RemoveContainer" containerID="cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca" Nov 03 13:16:58.070233 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:16:58.070212 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca\": container with ID starting with cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca not found: ID does not exist" containerID="cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca" Nov 03 13:16:58.070301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.070241 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca"} err="failed to get container status \"cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca\": rpc error: code = NotFound desc = could not find container \"cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca\": container with ID starting with cb72917762202f69a5d5cdf2dfbad511a857b75d772f6282449f5c409fe08fca not found: ID does not exist" Nov 03 13:16:58.214911 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.214873 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz"] Nov 03 13:16:58.261478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.261439 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cb1bd0b2-75cf85d899-mtmpz"] Nov 03 13:16:58.781653 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.781620 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:58.949863 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.949829 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e047289-890e-440c-8a63-e4561c401e9b" path="/var/lib/kubelet/pods/4e047289-890e-440c-8a63-e4561c401e9b/volumes" Nov 03 13:16:58.953389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:58.953365 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:59.067886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:59.067804 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:59.239230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:59.239196 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:59.416627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:59.416595 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:16:59.632421 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:16:59.632387 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:00.112559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:00.112529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:00.282706 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:00.282667 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:00.427283 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:00.427249 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:00.572577 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:00.572529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:00.929516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:00.929482 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:01.139396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:01.139322 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:01.845396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:01.845356 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:02.202745 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:02.202706 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:03.399413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:03.399381 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:03.580417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:03.580386 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:03.965588 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:03.965546 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:04.377122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:04.377015 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp"] Nov 03 13:17:04.522252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:04.522215 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:04.653456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:04.653372 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:05.707193 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:05.707158 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:05.915584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:05.915544 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:06.052025 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:06.051935 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:06.372943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:06.372798 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:06.565460 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:06.565429 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:06.702588 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:06.702556 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:09.377751 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:09.377707 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" podUID="80f50249-4e42-4751-8333-8d37fd843b7a" containerName="payload-processing" containerID="cri-o://7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067" gracePeriod=25 Nov 03 13:17:09.631604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:09.631543 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:17:09.787094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:09.787033 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f50249-4e42-4751-8333-8d37fd843b7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50249-4e42-4751-8333-8d37fd843b7a-plugins-config-volume\") pod \"80f50249-4e42-4751-8333-8d37fd843b7a\" (UID: \"80f50249-4e42-4751-8333-8d37fd843b7a\") " Nov 03 13:17:09.787094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:09.787098 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f50249-4e42-4751-8333-8d37fd843b7a-kube-api-access-9hjgr\" (UniqueName: \"kubernetes.io/projected/80f50249-4e42-4751-8333-8d37fd843b7a-kube-api-access-9hjgr\") pod \"80f50249-4e42-4751-8333-8d37fd843b7a\" (UID: \"80f50249-4e42-4751-8333-8d37fd843b7a\") " Nov 03 13:17:09.787455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:09.787426 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f50249-4e42-4751-8333-8d37fd843b7a-plugins-config-volume" pod "80f50249-4e42-4751-8333-8d37fd843b7a" (UID: "80f50249-4e42-4751-8333-8d37fd843b7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:09.789179 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:09.789157 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f50249-4e42-4751-8333-8d37fd843b7a-kube-api-access-9hjgr" pod "80f50249-4e42-4751-8333-8d37fd843b7a" (UID: "80f50249-4e42-4751-8333-8d37fd843b7a"). InnerVolumeSpecName "kube-api-access-9hjgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:09.887902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:09.887823 4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f50249-4e42-4751-8333-8d37fd843b7a-plugins-config-volume\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:09.887902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:09.887852 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hjgr\" (UniqueName: \"kubernetes.io/projected/80f50249-4e42-4751-8333-8d37fd843b7a-kube-api-access-9hjgr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:10.130306 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.130274 4924 generic.go:356] "Generic (PLEG): container finished" podID="80f50249-4e42-4751-8333-8d37fd843b7a" containerID="7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067" exitCode=0 Nov 03 13:17:10.130490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.130324 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" Nov 03 13:17:10.130490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.130357 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" event={"ID":"80f50249-4e42-4751-8333-8d37fd843b7a","Type":"ContainerDied","Data":"7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067"} Nov 03 13:17:10.130490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.130407 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp" event={"ID":"80f50249-4e42-4751-8333-8d37fd843b7a","Type":"ContainerDied","Data":"a0579d352aa26e4ef073b4ba9f3d45238edb2ae59710cedcfe5c16aede847e48"} Nov 03 13:17:10.130490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.130436 4924 scope.go:122] "RemoveContainer" containerID="7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067" Nov 03 13:17:10.139736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.139719 4924 scope.go:122] "RemoveContainer" containerID="7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067" Nov 03 13:17:10.139992 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:17:10.139976 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067\": container with ID starting with 7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067 not found: ID does not exist" containerID="7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067" Nov 03 13:17:10.140056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.140003 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067"} err="failed to get container status \"7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067\": rpc error: code = NotFound desc = could not find container \"7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067\": container with ID starting with 7071ca7a60e81abe79d59307d5dcdd57fc6d04ca97934270d13f1bdbe852c067 not found: ID does not exist" Nov 03 13:17:10.285561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.285521 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp"] Nov 03 13:17:10.331416 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.331379 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a7a9081d-6c989795d9-fq9qp"] Nov 03 13:17:10.949503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:10.949472 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f50249-4e42-4751-8333-8d37fd843b7a" path="/var/lib/kubelet/pods/80f50249-4e42-4751-8333-8d37fd843b7a/volumes" Nov 03 13:17:11.544920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:11.544882 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:11.714263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:11.714230 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:11.825262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:11.825190 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:12.028775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:12.028742 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:12.202389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:12.202359 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:13.499649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:13.499614 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:13.817813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:13.817728 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:14.152904 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:14.152807 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:14.579040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:14.579008 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:14.747546 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:14.747488 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:15.007510 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:15.007478 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:15.644855 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:15.644820 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:15.799136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:15.799101 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:17.370600 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:17.370563 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:18.559146 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:17:18.559110 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:17:19.678921 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:19.678885 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:19.862889 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:19.862850 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:19.997377 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:19.997297 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:20.764997 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:20.764963 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:20.897191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:20.897160 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:21.025177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:21.025045 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:23.116842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:23.116809 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:23.290241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:23.290207 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:23.559981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:23.559947 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:28.501427 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:28.501397 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:28.663028 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:28.662993 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:28.782539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:28.782462 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:29.033245 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:29.033160 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:29.214688 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:29.214652 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:33.199708 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.199666 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt"] Nov 03 13:17:33.200653 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.200617 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" podUID="f46e42bf-8406-4111-988b-2a898926cc1a" containerName="maas-api" containerID="cri-o://1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23" gracePeriod=30 Nov 03 13:17:33.289269 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.289236 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:33.478615 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.478583 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:17:33.600618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.600582 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-metrics-certs\") pod \"f46e42bf-8406-4111-988b-2a898926cc1a\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " Nov 03 13:17:33.600809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.600640 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls\") pod \"f46e42bf-8406-4111-988b-2a898926cc1a\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " Nov 03 13:17:33.600809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.600690 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46e42bf-8406-4111-988b-2a898926cc1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e42bf-8406-4111-988b-2a898926cc1a-openshift-service-ca\") pod \"f46e42bf-8406-4111-988b-2a898926cc1a\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " Nov 03 13:17:33.600809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.600740 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46e42bf-8406-4111-988b-2a898926cc1a-kube-api-access-dzzb7\" (UniqueName: \"kubernetes.io/projected/f46e42bf-8406-4111-988b-2a898926cc1a-kube-api-access-dzzb7\") pod \"f46e42bf-8406-4111-988b-2a898926cc1a\" (UID: \"f46e42bf-8406-4111-988b-2a898926cc1a\") " Nov 03 13:17:33.601108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.601061 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46e42bf-8406-4111-988b-2a898926cc1a-openshift-service-ca" pod "f46e42bf-8406-4111-988b-2a898926cc1a" (UID: "f46e42bf-8406-4111-988b-2a898926cc1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:33.602903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.602881 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-metrics-certs" pod "f46e42bf-8406-4111-988b-2a898926cc1a" (UID: "f46e42bf-8406-4111-988b-2a898926cc1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:17:33.603020 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.602928 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46e42bf-8406-4111-988b-2a898926cc1a-kube-api-access-dzzb7" pod "f46e42bf-8406-4111-988b-2a898926cc1a" (UID: "f46e42bf-8406-4111-988b-2a898926cc1a"). InnerVolumeSpecName "kube-api-access-dzzb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:33.603114 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.603098 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls" pod "f46e42bf-8406-4111-988b-2a898926cc1a" (UID: "f46e42bf-8406-4111-988b-2a898926cc1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:17:33.702272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.702232 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:33.702272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.702266 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46e42bf-8406-4111-988b-2a898926cc1a-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:33.702272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.702276 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46e42bf-8406-4111-988b-2a898926cc1a-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:33.702523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:33.702285 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzzb7\" (UniqueName: \"kubernetes.io/projected/f46e42bf-8406-4111-988b-2a898926cc1a-kube-api-access-dzzb7\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:34.273342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.273308 4924 generic.go:356] "Generic (PLEG): container finished" podID="f46e42bf-8406-4111-988b-2a898926cc1a" containerID="1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23" exitCode=0 Nov 03 13:17:34.273811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.273354 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" Nov 03 13:17:34.273811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.273371 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" event={"ID":"f46e42bf-8406-4111-988b-2a898926cc1a","Type":"ContainerDied","Data":"1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23"} Nov 03 13:17:34.273811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.273414 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt" event={"ID":"f46e42bf-8406-4111-988b-2a898926cc1a","Type":"ContainerDied","Data":"863b68561516dc917ccfc583f8fd7308028f322f22d49056576f94b53f69d2c1"} Nov 03 13:17:34.273811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.273433 4924 scope.go:122] "RemoveContainer" containerID="1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23" Nov 03 13:17:34.283318 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.283299 4924 scope.go:122] "RemoveContainer" containerID="1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23" Nov 03 13:17:34.283595 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:17:34.283576 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23\": container with ID starting with 1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23 not found: ID does not exist" containerID="1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23" Nov 03 13:17:34.283640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.283605 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23"} err="failed to get container status \"1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23\": rpc error: code = NotFound desc = could not find container \"1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23\": container with ID starting with 1bdcba513f6823e3f26a296b71d9b1e1d225d9acf330be091d9b0c75c9f48e23 not found: ID does not exist" Nov 03 13:17:34.427694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.427658 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt"] Nov 03 13:17:34.474128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.474094 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-11ca03ad-6f4fd4865d-rftqt"] Nov 03 13:17:34.946150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.946110 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:34.949764 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:34.949740 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46e42bf-8406-4111-988b-2a898926cc1a" path="/var/lib/kubelet/pods/f46e42bf-8406-4111-988b-2a898926cc1a/volumes" Nov 03 13:17:35.229848 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:35.229739 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:35.346550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:35.346517 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:36.048575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:36.048539 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:36.887170 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:36.887135 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:37.387663 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.387625 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr"] Nov 03 13:17:37.388567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388525 4924 cpu_manager.go:414] "containerMap: removing container" podUID="4e047289-890e-440c-8a63-e4561c401e9b" containerName="payload-pre-processing" Nov 03 13:17:37.388567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388554 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e047289-890e-440c-8a63-e4561c401e9b" containerName="payload-pre-processing" Nov 03 13:17:37.388567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388573 4924 cpu_manager.go:414] "containerMap: removing container" podUID="80f50249-4e42-4751-8333-8d37fd843b7a" containerName="payload-processing" Nov 03 13:17:37.388821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388581 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f50249-4e42-4751-8333-8d37fd843b7a" containerName="payload-processing" Nov 03 13:17:37.388821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388618 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f46e42bf-8406-4111-988b-2a898926cc1a" containerName="maas-api" Nov 03 13:17:37.388821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388626 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46e42bf-8406-4111-988b-2a898926cc1a" containerName="maas-api" Nov 03 13:17:37.388821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388748 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46e42bf-8406-4111-988b-2a898926cc1a" containerName="maas-api" Nov 03 13:17:37.388821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388768 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e047289-890e-440c-8a63-e4561c401e9b" containerName="payload-pre-processing" Nov 03 13:17:37.388821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.388779 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f50249-4e42-4751-8333-8d37fd843b7a" containerName="payload-processing" Nov 03 13:17:37.394649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.394620 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.447359 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.447328 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42c707-openshift-default-dockercfg-7t7mv\"" Nov 03 13:17:37.494264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.494232 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:37.496007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.495984 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr"] Nov 03 13:17:37.540119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540066 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.540304 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540183 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.540304 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540217 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.540304 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540270 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.540462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540443 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.540527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540492 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjbpq\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-kube-api-access-cjbpq\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.540673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540651 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.540731 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540691 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.540785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.540760 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.641922 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642234 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642275 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642331 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642376 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjbpq\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-kube-api-access-cjbpq\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642436 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642471 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642530 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642571 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642577 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642612 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642437 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.642981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.642956 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.643301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.643282 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.645302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.645280 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.645616 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.645594 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.659958 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.659930 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:37.695809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.695774 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.696463 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.696436 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjbpq\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-kube-api-access-cjbpq\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:37.713035 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:37.713001 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:38.131629 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:38.131602 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr"] Nov 03 13:17:38.133103 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:17:38.133050 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabbd2d2_2c43_4d4d_aeba_b687168041f0.slice/crio-b5ddc6452653c7c063809a2c278f0b7b094d88c15fe9282c89214e75bb10f44b WatchSource:0}: Error finding container b5ddc6452653c7c063809a2c278f0b7b094d88c15fe9282c89214e75bb10f44b: Status 404 returned error can't find the container with id b5ddc6452653c7c063809a2c278f0b7b094d88c15fe9282c89214e75bb10f44b Nov 03 13:17:38.135293 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:38.135258 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:17:38.135366 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:38.135321 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:17:38.135366 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:38.135351 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:17:38.296511 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:38.296465 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" event={"ID":"cabbd2d2-2c43-4d4d-aeba-b687168041f0","Type":"ContainerStarted","Data":"089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba"} Nov 03 13:17:38.296665 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:38.296542 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" event={"ID":"cabbd2d2-2c43-4d4d-aeba-b687168041f0","Type":"ContainerStarted","Data":"b5ddc6452653c7c063809a2c278f0b7b094d88c15fe9282c89214e75bb10f44b"} Nov 03 13:17:38.407119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:38.407051 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" podStartSLOduration=1.407036351 podStartE2EDuration="1.407036351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:38.405747009 +0000 UTC m=+791.600150602" watchObservedRunningTime="2026-11-03 13:17:38.407036351 +0000 UTC m=+791.601439945" Nov 03 13:17:38.714189 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:38.714111 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:39.718547 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:39.718519 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:40.306455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:40.306412 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:40.307516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:40.307495 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:17:41.066593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:41.066561 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:41.406785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:41.406600 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:41.601298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:41.601252 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:41.733606 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:41.733520 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:44.271462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:44.271430 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:44.900160 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:44.900128 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:45.030513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:45.030476 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:45.175774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:45.175740 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:46.349733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.349701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:46.522492 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.522457 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:46.591697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.591666 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6"] Nov 03 13:17:46.595601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.595578 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.691232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.691179 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-34b13c-serving-cert\"" Nov 03 13:17:46.691232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.691212 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgx9m\"" Nov 03 13:17:46.691232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.691213 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:17:46.694618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.694584 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6"] Nov 03 13:17:46.730135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.730095 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bljd\" (UniqueName: \"kubernetes.io/projected/9839ceb3-a0c7-4748-9f6b-307edd6bf097-kube-api-access-5bljd\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.730135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.730134 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9839ceb3-a0c7-4748-9f6b-307edd6bf097-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.730332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.730302 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-metrics-certs\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.730368 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.730336 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.800268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.800225 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:46.831233 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.831193 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-metrics-certs\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.831458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.831244 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.831458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.831271 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bljd\" (UniqueName: \"kubernetes.io/projected/9839ceb3-a0c7-4748-9f6b-307edd6bf097-kube-api-access-5bljd\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.831458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.831315 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9839ceb3-a0c7-4748-9f6b-307edd6bf097-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.831458 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:17:46.831394 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-serving-cert: secret "maas-api-e2e-worker-w1-34b13c-serving-cert" not found Nov 03 13:17:46.831688 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:17:46.831465 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls podName:9839ceb3-a0c7-4748-9f6b-307edd6bf097 nodeName:}" failed. No retries permitted until 2026-11-03 13:17:47.331444994 +0000 UTC m=+800.525848584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls") pod "maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" (UID: "9839ceb3-a0c7-4748-9f6b-307edd6bf097") : secret "maas-api-e2e-worker-w1-34b13c-serving-cert" not found Nov 03 13:17:46.832037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.832014 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9839ceb3-a0c7-4748-9f6b-307edd6bf097-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.833839 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.833813 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-metrics-certs\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.887422 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.887316 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bljd\" (UniqueName: \"kubernetes.io/projected/9839ceb3-a0c7-4748-9f6b-307edd6bf097-kube-api-access-5bljd\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:46.987001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:46.986965 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:47.124556 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:47.124526 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:47.336479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:47.336437 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:47.338834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:47.338811 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls\") pod \"maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:47.508842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:47.508806 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:47.890638 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:47.890603 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6"] Nov 03 13:17:47.891732 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:17:47.891703 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9839ceb3_a0c7_4748_9f6b_307edd6bf097.slice/crio-56b0682609f8093f43a0b6b282a5e733f094bb17420d718771f77e55611cde55 WatchSource:0}: Error finding container 56b0682609f8093f43a0b6b282a5e733f094bb17420d718771f77e55611cde55: Status 404 returned error can't find the container with id 56b0682609f8093f43a0b6b282a5e733f094bb17420d718771f77e55611cde55 Nov 03 13:17:48.351151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:48.351100 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" event={"ID":"9839ceb3-a0c7-4748-9f6b-307edd6bf097","Type":"ContainerStarted","Data":"56b0682609f8093f43a0b6b282a5e733f094bb17420d718771f77e55611cde55"} Nov 03 13:17:48.564665 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:17:48.564624 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:17:48.605218 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:48.605134 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:49.357289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:49.357252 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" event={"ID":"9839ceb3-a0c7-4748-9f6b-307edd6bf097","Type":"ContainerStarted","Data":"d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4"} Nov 03 13:17:49.357497 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:49.357318 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:49.466645 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:49.466592 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" podStartSLOduration=2.571550554 podStartE2EDuration="3.466577282s" totalImagesPullingTime="895.026728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:49.465766446 +0000 UTC m=+802.660170041" watchObservedRunningTime="2026-11-03 13:17:49.466577282 +0000 UTC m=+802.660980873" Nov 03 13:17:52.197276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:52.197239 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:17:55.369258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:55.369223 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 13:17:57.477521 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:17:57.477470 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:01.494296 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:01.494249 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:02.120695 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:02.120662 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:02.249460 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:02.249425 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:02.381156 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:02.381062 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:03.183150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:03.183110 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:06.709304 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:06.709268 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:07.300932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:07.300892 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:07.422535 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:07.422501 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:07.544670 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:07.544629 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:08.423579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.423545 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd"] Nov 03 13:18:08.427774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.427744 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.450596 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.450559 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:08.515548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.515511 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd"] Nov 03 13:18:08.523809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.523778 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cdb532-openshift-default-dockercfg-q2fdz\"" Nov 03 13:18:08.537858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.537817 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.538033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.537877 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.538148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.538023 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590429ba-2e0e-44c7-9635-4b60db71b247-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.538148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.538059 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590429ba-2e0e-44c7-9635-4b60db71b247-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.538238 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.538149 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.538273 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.538233 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.538305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.538271 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.538393 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.538374 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fv8d\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-kube-api-access-4fv8d\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.538462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.538424 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.614143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.614110 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:08.639285 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639253 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639305 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590429ba-2e0e-44c7-9635-4b60db71b247-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639321 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590429ba-2e0e-44c7-9635-4b60db71b247-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639428 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639578 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639484 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639578 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639500 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639578 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639524 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fv8d\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-kube-api-access-4fv8d\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639569 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639640 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639835 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639797 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639881 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.639989 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.639953 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.640133 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.640109 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.640183 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.640114 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590429ba-2e0e-44c7-9635-4b60db71b247-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.641779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.641758 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590429ba-2e0e-44c7-9635-4b60db71b247-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.641926 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.641906 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.692957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.692887 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.693111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.693000 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fv8d\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-kube-api-access-4fv8d\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:08.732421 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.732385 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:08.744450 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:08.744423 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:09.161765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:09.161721 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd"] Nov 03 13:18:09.198240 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:18:09.198208 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590429ba_2e0e_44c7_9635_4b60db71b247.slice/crio-0e4e5d32806e35584c1dacc0e5abe08bc7f21e9788c5fea6831642908bc63ea6 WatchSource:0}: Error finding container 0e4e5d32806e35584c1dacc0e5abe08bc7f21e9788c5fea6831642908bc63ea6: Status 404 returned error can't find the container with id 0e4e5d32806e35584c1dacc0e5abe08bc7f21e9788c5fea6831642908bc63ea6 Nov 03 13:18:09.200314 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:09.200278 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:18:09.200390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:09.200353 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:18:09.200390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:09.200383 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:18:09.466182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:09.466097 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" event={"ID":"590429ba-2e0e-44c7-9635-4b60db71b247","Type":"ContainerStarted","Data":"b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea"} Nov 03 13:18:09.466182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:09.466159 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" event={"ID":"590429ba-2e0e-44c7-9635-4b60db71b247","Type":"ContainerStarted","Data":"0e4e5d32806e35584c1dacc0e5abe08bc7f21e9788c5fea6831642908bc63ea6"} Nov 03 13:18:09.575674 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:09.575615 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" podStartSLOduration=1.575601614 podStartE2EDuration="1.575601614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:09.574347975 +0000 UTC m=+822.768751568" watchObservedRunningTime="2026-11-03 13:18:09.575601614 +0000 UTC m=+822.770005208" Nov 03 13:18:09.745317 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:09.745165 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:10.751638 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:10.750160 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 13:18:10.751638 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:10.750238 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 13:18:11.748095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:11.747443 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 13:18:11.748095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:11.747511 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 13:18:12.748093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:12.745975 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 13:18:12.748093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:12.746129 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 13:18:13.083413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:13.083327 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:13.746129 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:13.745508 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 13:18:13.746129 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:13.745587 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 13:18:13.863030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:13.862981 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:14.750096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:14.746438 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 13:18:14.750096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:14.746523 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 13:18:15.749469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:15.749436 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:15.749854 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:15.749833 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:15.750433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:15.750415 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:18:18.578524 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:18:18.578479 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:18:19.061597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:19.061557 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:21.269732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:21.269690 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:24.169245 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:24.169213 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:25.695188 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:25.695154 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:26.301819 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:26.301779 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:26.347442 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:26.347402 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:18:26.347608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:26.347479 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:18:26.453592 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:26.453537 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:26.781438 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:26.781381 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:26.924254 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:26.924220 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:27.607888 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:27.607855 4924 generic.go:356] "Generic (PLEG): container finished" podID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerID="089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba" exitCode=0 Nov 03 13:18:27.608154 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:27.607932 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" event={"ID":"cabbd2d2-2c43-4d4d-aeba-b687168041f0","Type":"ContainerDied","Data":"089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba"} Nov 03 13:18:27.608397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:27.608380 4924 scope.go:122] "RemoveContainer" containerID="089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba" Nov 03 13:18:27.609716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:27.609686 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:18:27.609795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:27.609753 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:18:27.609795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:27.609786 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:18:28.615047 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:28.615008 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" event={"ID":"cabbd2d2-2c43-4d4d-aeba-b687168041f0","Type":"ContainerStarted","Data":"e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c"} Nov 03 13:18:28.713410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:28.713371 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:18:28.713617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:28.713559 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:18:28.713694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:28.713608 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 13:18:29.660731 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:29.660701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:29.713790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:29.713759 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:18:29.713965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:29.713845 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 13:18:30.713814 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:30.713775 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:18:30.714229 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:30.713830 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 13:18:31.714138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:31.714046 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:18:31.714138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:31.714120 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 13:18:32.713410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:32.713372 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:18:32.713599 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:32.713434 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 13:18:33.728389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:33.728359 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:18:33.728896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:33.728734 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:18:33.729725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:33.729701 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 13:18:34.787932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:34.787895 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:39.866593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:39.866513 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:40.249520 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:40.249484 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:45.148922 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:45.148883 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:47.043204 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.043166 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw"] Nov 03 13:18:47.045936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.045914 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.131965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.131930 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:47.137220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.137189 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ptk\" (UniqueName: \"kubernetes.io/projected/b0b82e11-cb81-4df8-8d5e-65398626aaae-kube-api-access-29ptk\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.137411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.137392 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-dshm\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.137502 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.137471 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b82e11-cb81-4df8-8d5e-65398626aaae-tls-certs\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.137641 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.137612 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-tmp-dir\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.137757 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.137743 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-model-cache\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.137813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.137779 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-home\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.141328 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.141302 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"simulator-w1-34b13c-kserve-self-signed-certs\"" Nov 03 13:18:47.141452 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.141396 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"default-dockercfg-jc47d\"" Nov 03 13:18:47.141519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.141474 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"openshift-service-ca.crt\"" Nov 03 13:18:47.141595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.141574 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"kube-root-ca.crt\"" Nov 03 13:18:47.144401 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.144377 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw"] Nov 03 13:18:47.238797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.238765 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-model-cache\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.238991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.238803 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-home\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.238991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.238825 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ptk\" (UniqueName: \"kubernetes.io/projected/b0b82e11-cb81-4df8-8d5e-65398626aaae-kube-api-access-29ptk\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.238991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.238843 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-dshm\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.238991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.238890 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b82e11-cb81-4df8-8d5e-65398626aaae-tls-certs\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.238991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.238930 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-tmp-dir\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.239278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.239258 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-home\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.239522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.239503 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-model-cache\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.239621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.239604 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-tmp-dir\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.241256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.241236 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-dshm\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.241668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.241648 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b82e11-cb81-4df8-8d5e-65398626aaae-tls-certs\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.305206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.305132 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ptk\" (UniqueName: \"kubernetes.io/projected/b0b82e11-cb81-4df8-8d5e-65398626aaae-kube-api-access-29ptk\") pod \"simulator-w1-34b13c-kserve-8564c4df6d-h4tcw\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.332196 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.332160 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:47.372445 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.372402 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:47.744573 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.744531 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw"] Nov 03 13:18:47.746408 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:18:47.746379 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0b82e11_cb81_4df8_8d5e_65398626aaae.slice/crio-97f901397fad905a08370506346903461e0bf35e71ca81455a17360d5845107f WatchSource:0}: Error finding container 97f901397fad905a08370506346903461e0bf35e71ca81455a17360d5845107f: Status 404 returned error can't find the container with id 97f901397fad905a08370506346903461e0bf35e71ca81455a17360d5845107f Nov 03 13:18:47.912291 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:47.912256 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:48.059151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:48.058672 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:48.191106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:48.191033 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:48.558278 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:18:48.558241 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:18:48.758948 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:48.758911 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" event={"ID":"b0b82e11-cb81-4df8-8d5e-65398626aaae","Type":"ContainerStarted","Data":"97f901397fad905a08370506346903461e0bf35e71ca81455a17360d5845107f"} Nov 03 13:18:49.767332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:49.767290 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" event={"ID":"b0b82e11-cb81-4df8-8d5e-65398626aaae","Type":"ContainerStarted","Data":"92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752"} Nov 03 13:18:49.880310 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:49.880194 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" podStartSLOduration=1.06230798 podStartE2EDuration="2.880179563s" totalImagesPullingTime="1.817871583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:49.879697946 +0000 UTC m=+863.074101539" watchObservedRunningTime="2026-11-03 13:18:49.880179563 +0000 UTC m=+863.074583156" Nov 03 13:18:50.373128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:50.373093 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:50.560001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:50.559967 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:55.657749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:55.657690 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:18:57.372738 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:57.372696 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:57.385623 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:57.385589 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:18:57.821139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:18:57.821107 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:19:00.812402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:00.812361 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:05.876986 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:05.876953 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:07.097085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:07.097039 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:11.073650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:11.073612 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:16.184281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:16.184242 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:18.558227 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:19:18.558192 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:19:21.358956 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:21.358922 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:21.641334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:21.641252 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:22.131624 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:22.131593 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:26.465963 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:26.465922 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:27.056704 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:27.056673 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:19:27.057805 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:27.057757 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:19:30.437524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:30.437493 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:31.728380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:31.728346 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:36.774611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:36.774572 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:41.974886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:41.974818 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:47.185101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:47.185045 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:48.559003 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:19:48.558969 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:19:51.642730 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:51.642698 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:52.474436 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:52.474404 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:19:57.567661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:19:57.567631 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:02.688718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:02.688679 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:07.762656 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:07.762584 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:12.866447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:12.866417 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:17.972844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:17.972810 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:18.558407 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:20:18.558372 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:20:19.929108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:19.929034 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:21.612054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.612019 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j"] Nov 03 13:20:21.616164 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.616141 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.682301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.682265 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:21.703985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.703954 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-dshm\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.704157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.704062 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tls-certs\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.704157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.704135 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm2pk\" (UniqueName: \"kubernetes.io/projected/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-kube-api-access-rm2pk\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.704234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.704195 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-model-cache\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.704276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.704235 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tmp-dir\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.704276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.704253 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-home\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.712756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.712734 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"openshift-service-ca.crt\"" Nov 03 13:20:21.712756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.712734 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"kube-root-ca.crt\"" Nov 03 13:20:21.712981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.712837 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"default-dockercfg-mzpk2\"" Nov 03 13:20:21.712981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.712899 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"premium-w0-42c707-kserve-self-signed-certs\"" Nov 03 13:20:21.715402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.715384 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j"] Nov 03 13:20:21.805476 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.805441 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-model-cache\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.805648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.805499 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tmp-dir\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.805648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.805519 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-home\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.805648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.805553 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-dshm\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.805648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.805630 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tls-certs\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.805872 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.805684 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm2pk\" (UniqueName: \"kubernetes.io/projected/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-kube-api-access-rm2pk\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.805975 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.805954 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tmp-dir\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.806045 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.805996 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-home\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.806200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.806175 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-model-cache\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.808128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.808096 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-dshm\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.808255 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.808239 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tls-certs\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.810718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.810690 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:21.861017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.860982 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm2pk\" (UniqueName: \"kubernetes.io/projected/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-kube-api-access-rm2pk\") pod \"premium-w0-42c707-kserve-cf8fcc897-4kg4j\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.928545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.928508 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:21.953486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:21.953444 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:22.324368 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:22.324347 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j"] Nov 03 13:20:22.326271 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:20:22.326245 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0494ee2_8d1e_4df7_b26e_8be68d3d1c42.slice/crio-ecec620ed3af730a583c87a3d994020ae4f2187a278e41d451d66b884b070103 WatchSource:0}: Error finding container ecec620ed3af730a583c87a3d994020ae4f2187a278e41d451d66b884b070103: Status 404 returned error can't find the container with id ecec620ed3af730a583c87a3d994020ae4f2187a278e41d451d66b884b070103 Nov 03 13:20:23.159261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:23.159231 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:23.326376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:23.326342 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" event={"ID":"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42","Type":"ContainerStarted","Data":"82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531"} Nov 03 13:20:23.326560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:23.326384 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" event={"ID":"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42","Type":"ContainerStarted","Data":"ecec620ed3af730a583c87a3d994020ae4f2187a278e41d451d66b884b070103"} Nov 03 13:20:23.436860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:23.436756 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" podStartSLOduration=2.436739191 podStartE2EDuration="2.436739191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:23.435513875 +0000 UTC m=+956.629917459" watchObservedRunningTime="2026-11-03 13:20:23.436739191 +0000 UTC m=+956.631142786" Nov 03 13:20:23.929557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:23.929525 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:28.312547 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:28.312514 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:31.929483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:31.929445 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:31.942562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:31.942515 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:32.394166 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:32.394063 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 13:20:33.373755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:33.373716 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:38.522661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:38.522630 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:43.590696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:43.590658 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:48.556955 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:20:48.556918 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:20:48.645631 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:48.645600 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:53.775844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:53.775804 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:55.327851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:55.327814 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:55.451734 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:55.451700 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:20:58.857866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:20:58.857833 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:01.284256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:01.284194 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:01.429455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:01.429424 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:01.561508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:01.561413 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:04.004465 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:04.004427 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:05.598530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:05.598494 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:09.126647 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:09.126593 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:14.303780 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:14.303716 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:17.573026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.572979 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2"] Nov 03 13:21:17.577453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.577433 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649127 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649091 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649127 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649131 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649230 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649295 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqcc\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-kube-api-access-jvqcc\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649365 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649535 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649423 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649535 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649465 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649535 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649515 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.649718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.649692 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.673267 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.673234 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-34b13c-openshift-default-dockercfg-kf4m8\"" Nov 03 13:21:17.676672 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.676647 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2"] Nov 03 13:21:17.750769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.750732 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.750980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.750791 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.750980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.750816 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.750980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.750862 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.750980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.750904 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqcc\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-kube-api-access-jvqcc\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.750980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.750932 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.750980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.750980 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.751331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.751022 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.751331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.751094 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.751331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.751260 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.751488 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.751421 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.751568 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.751547 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.752321 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.752267 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.752429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.752365 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.753647 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.753628 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.753984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.753967 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.806002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.805969 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.807094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.807044 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqcc\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-kube-api-access-jvqcc\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:17.891209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:17.891114 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:18.309924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:18.309896 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2"] Nov 03 13:21:18.311342 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:21:18.311312 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ea1109_ec3c_4f0f_9d5e_1566441f2b1e.slice/crio-c653b2f938377e113dbee2d0f591ada6ccfda43602bda6e97403cbd7c049ad6e WatchSource:0}: Error finding container c653b2f938377e113dbee2d0f591ada6ccfda43602bda6e97403cbd7c049ad6e: Status 404 returned error can't find the container with id c653b2f938377e113dbee2d0f591ada6ccfda43602bda6e97403cbd7c049ad6e Nov 03 13:21:18.313663 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:18.313627 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:18.313773 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:18.313702 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:18.313773 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:18.313750 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:18.559653 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:21:18.559564 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:21:18.663145 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:18.663105 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" event={"ID":"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e","Type":"ContainerStarted","Data":"e6e51b42a37437b2514e2187824e01d89370407bcd41bb70ce0bae106cf68896"} Nov 03 13:21:18.663546 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:18.663171 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" event={"ID":"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e","Type":"ContainerStarted","Data":"c653b2f938377e113dbee2d0f591ada6ccfda43602bda6e97403cbd7c049ad6e"} Nov 03 13:21:18.775938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:18.775876 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" podStartSLOduration=1.775861875 podStartE2EDuration="1.775861875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:18.773313438 +0000 UTC m=+1011.967717032" watchObservedRunningTime="2026-11-03 13:21:18.775861875 +0000 UTC m=+1011.970265468" Nov 03 13:21:18.891518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:18.891438 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:19.485266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:19.485150 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:19.633619 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:19.633582 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:19.894016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:19.893914 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:21:19.894016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:19.893968 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:21:20.898098 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:20.896535 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:21:20.898098 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:20.896598 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:21:21.762618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:21.762527 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:21.894095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:21.892602 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:21:21.894095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:21.892680 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:21:21.934293 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:21.934157 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:21.934293 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:21.934221 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:22.321630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:22.321587 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:22.892005 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:22.891965 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:21:22.892235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:22.892022 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:21:23.891716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:23.891669 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:21:23.892256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:23.891729 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:21:24.775838 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:24.775798 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:25.769236 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:25.769204 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:25.769668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:25.769625 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:25.770403 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:25.770385 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 13:21:29.880690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:29.880656 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:35.452474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:35.452431 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:36.921277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:36.921187 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:36.921672 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:36.921281 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:40.660616 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:40.660569 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:43.173646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.173609 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc"] Nov 03 13:21:43.177332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.177307 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.220014 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.219978 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb"] Nov 03 13:21:43.223683 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.223662 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.268986 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.268946 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.269150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269011 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269061 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.269225 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269173 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.269225 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269208 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b31c398-2145-4e13-a73e-5715bdce2ae4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269324 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269307 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7lgc\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-kube-api-access-h7lgc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.269360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269338 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269422 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269407 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.269461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269437 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhwm4\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-kube-api-access-hhwm4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269517 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269495 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.269565 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269528 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269565 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269558 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.269686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269662 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269711 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6979f54-42fc-4c34-9a00-f718a1d04297-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.269746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269736 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269852 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269788 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269840 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.269898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.269894 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.275387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.275361 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc"] Nov 03 13:21:43.321716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.321677 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb"] Nov 03 13:21:43.371247 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371204 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371260 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371356 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.371437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371387 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371404 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371420 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b31c398-2145-4e13-a73e-5715bdce2ae4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.371715 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371452 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7lgc\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-kube-api-access-h7lgc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371739 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.371825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371771 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371794 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371770 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371822 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.371984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371827 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhwm4\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-kube-api-access-hhwm4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.371984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371819 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371899 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.371984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371954 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.371995 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372032 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372096 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6979f54-42fc-4c34-9a00-f718a1d04297-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372105 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372122 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372105 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372165 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372200 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372233 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b31c398-2145-4e13-a73e-5715bdce2ae4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372814 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372418 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372814 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372626 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.372936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.372864 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6979f54-42fc-4c34-9a00-f718a1d04297-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.374180 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.374155 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.374537 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.374514 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.374698 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.374677 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.374818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.374802 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.431832 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.431768 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.432236 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.432211 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7lgc\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-kube-api-access-h7lgc\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-86scc\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.436362 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.436330 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.437202 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.437182 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhwm4\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-kube-api-access-hhwm4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j6nfb\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.490137 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.490102 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:43.536332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.536298 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:43.906767 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.906736 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc"] Nov 03 13:21:43.907263 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:21:43.907235 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6979f54_42fc_4c34_9a00_f718a1d04297.slice/crio-978239a19b8485da45b0664d8dda6ec0953912c1936f2c555b8366c00bf32966 WatchSource:0}: Error finding container 978239a19b8485da45b0664d8dda6ec0953912c1936f2c555b8366c00bf32966: Status 404 returned error can't find the container with id 978239a19b8485da45b0664d8dda6ec0953912c1936f2c555b8366c00bf32966 Nov 03 13:21:43.909567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.909524 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:43.909673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.909600 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:43.909673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.909645 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:43.955725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.955700 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb"] Nov 03 13:21:43.956618 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:21:43.956593 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b31c398_2145_4e13_a73e_5715bdce2ae4.slice/crio-92fe56a364fb2c2d41f3cb20c6ac68019775fad06cf01edf3dc4b623841966c0 WatchSource:0}: Error finding container 92fe56a364fb2c2d41f3cb20c6ac68019775fad06cf01edf3dc4b623841966c0: Status 404 returned error can't find the container with id 92fe56a364fb2c2d41f3cb20c6ac68019775fad06cf01edf3dc4b623841966c0 Nov 03 13:21:43.959136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.959105 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:43.959209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.959168 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:43.959209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:43.959198 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:44.881096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:44.880612 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" event={"ID":"f6979f54-42fc-4c34-9a00-f718a1d04297","Type":"ContainerStarted","Data":"d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8"} Nov 03 13:21:44.881096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:44.880699 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" event={"ID":"f6979f54-42fc-4c34-9a00-f718a1d04297","Type":"ContainerStarted","Data":"978239a19b8485da45b0664d8dda6ec0953912c1936f2c555b8366c00bf32966"} Nov 03 13:21:44.887336 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:44.887296 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" event={"ID":"2b31c398-2145-4e13-a73e-5715bdce2ae4","Type":"ContainerStarted","Data":"f106b12c01ccf79ceca11f23c86fbeeeefa2dfed8a8d1f82c968ac782ad31a55"} Nov 03 13:21:44.887499 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:44.887366 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" event={"ID":"2b31c398-2145-4e13-a73e-5715bdce2ae4","Type":"ContainerStarted","Data":"92fe56a364fb2c2d41f3cb20c6ac68019775fad06cf01edf3dc4b623841966c0"} Nov 03 13:21:45.096113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.095360 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podStartSLOduration=2.095345877 podStartE2EDuration="2.095345877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:45.051389893 +0000 UTC m=+1038.245793488" watchObservedRunningTime="2026-11-03 13:21:45.095345877 +0000 UTC m=+1038.289749532" Nov 03 13:21:45.177095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.169817 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podStartSLOduration=2.169796854 podStartE2EDuration="2.169796854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:45.165524745 +0000 UTC m=+1038.359928342" watchObservedRunningTime="2026-11-03 13:21:45.169796854 +0000 UTC m=+1038.364200449" Nov 03 13:21:45.491061 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.490970 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:45.492848 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.492823 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:21:45.492965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.492863 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:21:45.543097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.542637 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:45.568097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.565228 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:45.568097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.565310 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:45.867131 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:45.866928 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:46.491421 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:46.491388 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:21:46.491785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:46.491449 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:21:46.549095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:46.548530 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:46.549095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:46.548601 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:47.491171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:47.491132 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:21:47.491341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:47.491193 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:21:47.545095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:47.538537 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:47.545095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:47.538603 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:48.491284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:48.491244 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:21:48.491471 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:48.491294 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:21:48.541097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:48.538483 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:48.541097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:48.538545 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:48.605100 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:21:48.605046 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:21:49.490602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:49.490562 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:21:49.490793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:49.490618 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:21:49.542097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:49.539568 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:49.542097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:49.539646 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:50.490937 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:50.490897 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:21:50.491345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:50.490966 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:21:50.536752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:50.536696 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:50.536993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:50.536768 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:50.972103 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:50.972046 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:21:51.490595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:51.490558 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:21:51.490774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:51.490620 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:21:51.537262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:51.537223 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:51.537897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:51.537865 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:52.706348 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:52.706316 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:52.707442 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:52.707417 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:52.709052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:52.709028 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:21:52.861517 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:52.861478 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:52.862095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:52.862061 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:52.862789 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:52.862762 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:21:56.078764 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:21:56.078729 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:00.742810 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:00.742769 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:01.197504 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:01.197470 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:04.457056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.457021 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2"] Nov 03 13:22:04.462779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.462755 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.536462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.536428 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:04.558432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.558399 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"distinct-w1-34b13c-kserve-self-signed-certs\"" Nov 03 13:22:04.561110 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.561083 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2"] Nov 03 13:22:04.644313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.644280 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tmp-dir\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.644513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.644340 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-model-cache\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.644513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.644482 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-dshm\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.644608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.644526 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-home\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.644643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.644610 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8jc\" (UniqueName: \"kubernetes.io/projected/077a2a59-e1a7-4f11-af5d-4373cc6643ad-kube-api-access-mx8jc\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.644688 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.644660 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tls-certs\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.745976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.745877 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-dshm\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.745976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.745939 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-home\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.746216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.745993 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8jc\" (UniqueName: \"kubernetes.io/projected/077a2a59-e1a7-4f11-af5d-4373cc6643ad-kube-api-access-mx8jc\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.746216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.746061 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tls-certs\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.746216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.746145 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tmp-dir\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.746336 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.746231 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-model-cache\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.746727 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.746486 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-home\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.746727 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.746625 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-model-cache\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.747164 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.747142 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tmp-dir\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.748756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.748725 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-dshm\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.749274 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.749252 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tls-certs\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:04.800552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:04.800515 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8jc\" (UniqueName: \"kubernetes.io/projected/077a2a59-e1a7-4f11-af5d-4373cc6643ad-kube-api-access-mx8jc\") pod \"distinct-w1-34b13c-kserve-855595cc76-vckl2\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:05.075197 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:05.075115 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:05.443625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:05.443595 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2"] Nov 03 13:22:05.444926 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:22:05.444893 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod077a2a59_e1a7_4f11_af5d_4373cc6643ad.slice/crio-451f2c9813db082dafcbd5e4255d4a0321beae61cdc3aaf963bf6e201fd941df WatchSource:0}: Error finding container 451f2c9813db082dafcbd5e4255d4a0321beae61cdc3aaf963bf6e201fd941df: Status 404 returned error can't find the container with id 451f2c9813db082dafcbd5e4255d4a0321beae61cdc3aaf963bf6e201fd941df Nov 03 13:22:06.021397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:06.021359 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" event={"ID":"077a2a59-e1a7-4f11-af5d-4373cc6643ad","Type":"ContainerStarted","Data":"1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c"} Nov 03 13:22:06.021397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:06.021401 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" event={"ID":"077a2a59-e1a7-4f11-af5d-4373cc6643ad","Type":"ContainerStarted","Data":"451f2c9813db082dafcbd5e4255d4a0321beae61cdc3aaf963bf6e201fd941df"} Nov 03 13:22:06.075631 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:06.075595 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:06.133163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:06.133108 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" podStartSLOduration=2.133093227 podStartE2EDuration="2.133093227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:06.130993389 +0000 UTC m=+1059.325397001" watchObservedRunningTime="2026-11-03 13:22:06.133093227 +0000 UTC m=+1059.327496824" Nov 03 13:22:06.280177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:06.280090 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:11.378673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:11.378630 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:15.076052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:15.076018 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:15.089202 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:15.089176 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:16.088484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:16.088454 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 13:22:16.469461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:16.469429 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:18.558206 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:22:18.558172 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:22:21.673312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:21.673263 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:22.675115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:22.675062 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:26.883017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:26.882933 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:27.844710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.844673 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp"] Nov 03 13:22:27.849691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.849667 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:27.927585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.927534 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:27.947259 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.947223 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"e2e-conflict-b-f2a72750-kserve-self-signed-certs\"" Nov 03 13:22:27.950128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.950105 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp"] Nov 03 13:22:27.996163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.996118 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tmp-dir\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:27.996354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.996193 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-home\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:27.996354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.996274 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhzd\" (UniqueName: \"kubernetes.io/projected/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-kube-api-access-xkhzd\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:27.996354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.996318 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-model-cache\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:27.996514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.996497 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tls-certs\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:27.996570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:27.996543 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-dshm\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.097122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097032 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhzd\" (UniqueName: \"kubernetes.io/projected/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-kube-api-access-xkhzd\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.097122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097090 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-model-cache\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.097360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097156 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tls-certs\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.097360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097192 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-dshm\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.097360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097226 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tmp-dir\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.097360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097295 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-home\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.097606 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097582 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-model-cache\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.097904 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097879 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-home\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.098066 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.097913 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tmp-dir\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.099617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.099588 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-dshm\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.099941 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.099921 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tls-certs\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.150919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.150885 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhzd\" (UniqueName: \"kubernetes.io/projected/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-kube-api-access-xkhzd\") pod \"e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.161419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.161379 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:28.551401 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:28.551369 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp"] Nov 03 13:22:28.552476 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:22:28.552445 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2b58384_2e0a_41e5_8a24_6f9fbcb449c3.slice/crio-487a80bda37832cb2177d143f465e1185448531d8e4e436ed129bd1781207608 WatchSource:0}: Error finding container 487a80bda37832cb2177d143f465e1185448531d8e4e436ed129bd1781207608: Status 404 returned error can't find the container with id 487a80bda37832cb2177d143f465e1185448531d8e4e436ed129bd1781207608 Nov 03 13:22:29.172331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:29.172291 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" event={"ID":"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3","Type":"ContainerStarted","Data":"61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9"} Nov 03 13:22:29.172716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:29.172344 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" event={"ID":"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3","Type":"ContainerStarted","Data":"487a80bda37832cb2177d143f465e1185448531d8e4e436ed129bd1781207608"} Nov 03 13:22:29.283598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:29.283546 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" podStartSLOduration=2.283531768 podStartE2EDuration="2.283531768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:29.280669246 +0000 UTC m=+1082.475072849" watchObservedRunningTime="2026-11-03 13:22:29.283531768 +0000 UTC m=+1082.477935361" Nov 03 13:22:30.162446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:30.162411 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:31.975790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:31.975757 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:34.393768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.393730 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj"] Nov 03 13:22:34.401407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.401368 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.450652 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.450622 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3c5dbc-openshift-default-dockercfg-hfbq4\"" Nov 03 13:22:34.500118 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.500085 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj"] Nov 03 13:22:34.572100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572041 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.572100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572105 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7e333-ae06-4712-acf2-792344bbd8eb-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.572309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572179 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.572309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572205 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.572309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572221 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.572309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572241 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.572454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572412 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkh6c\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-kube-api-access-qkh6c\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.572560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572536 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.572612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.572597 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674358 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674318 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674374 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7e333-ae06-4712-acf2-792344bbd8eb-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674411 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674449 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674481 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674509 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674571 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkh6c\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-kube-api-access-qkh6c\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674645 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674673 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.674829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.674804 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.675301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.675255 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.675424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.675327 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.675547 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.675521 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.675928 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.675896 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7e333-ae06-4712-acf2-792344bbd8eb-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.677040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.677015 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.677847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.677826 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.728239 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.728207 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:34.728504 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:34.728479 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkh6c\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-kube-api-access-qkh6c\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:35.016486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:35.016393 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:35.443006 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:35.442968 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj"] Nov 03 13:22:35.479015 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:22:35.478979 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb7e333_ae06_4712_acf2_792344bbd8eb.slice/crio-a6a42404470900097319c6bf14b0debfc649ca99a9fd48b73c2ee7f1f87f6af2 WatchSource:0}: Error finding container a6a42404470900097319c6bf14b0debfc649ca99a9fd48b73c2ee7f1f87f6af2: Status 404 returned error can't find the container with id a6a42404470900097319c6bf14b0debfc649ca99a9fd48b73c2ee7f1f87f6af2 Nov 03 13:22:35.481173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:35.481134 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:22:35.481253 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:35.481202 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:22:35.481253 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:35.481235 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:22:36.217952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:36.217908 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" event={"ID":"fbb7e333-ae06-4712-acf2-792344bbd8eb","Type":"ContainerStarted","Data":"767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42"} Nov 03 13:22:36.218198 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:36.217983 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" event={"ID":"fbb7e333-ae06-4712-acf2-792344bbd8eb","Type":"ContainerStarted","Data":"a6a42404470900097319c6bf14b0debfc649ca99a9fd48b73c2ee7f1f87f6af2"} Nov 03 13:22:36.381009 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:36.380952 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podStartSLOduration=2.380935248 podStartE2EDuration="2.380935248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:36.378130862 +0000 UTC m=+1089.572534457" watchObservedRunningTime="2026-11-03 13:22:36.380935248 +0000 UTC m=+1089.575338842" Nov 03 13:22:37.017617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:37.017580 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:37.019243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:37.019219 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:22:37.019303 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:37.019265 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 13:22:37.069051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:37.068994 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:38.016703 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:38.016661 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:22:38.017234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:38.016710 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 13:22:38.173368 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:38.173299 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:38.192886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:38.190720 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:38.315886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:38.315794 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:22:39.017509 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:39.017472 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:22:39.017686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:39.017529 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 13:22:40.017399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:40.017357 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:22:40.017776 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:40.017422 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 13:22:41.017365 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:41.017328 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:22:41.017538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:41.017396 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 13:22:42.017082 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:42.017040 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:22:42.017266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:42.017116 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 13:22:42.086562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:42.086515 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:43.021343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:43.021311 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:43.021834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:43.021814 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:43.022462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:43.022434 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 13:22:47.191720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:47.191668 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:48.557642 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:22:48.557597 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:22:52.262186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:52.262146 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:22:57.371184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:22:57.371148 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:02.473733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:02.473688 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:03.153516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:03.153482 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:07.787148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:07.786215 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:12.874255 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:12.874214 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:17.765011 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:17.764976 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:18.012869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:18.012820 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:18.559977 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:23:18.559943 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:23:23.000007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:22.999974 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:23.145559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:23.145527 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:28.275532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:28.275500 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:28.397212 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:28.397176 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:33.469605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:33.469564 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:38.570793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:38.570758 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:43.673796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:43.673756 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:48.561785 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:23:48.561742 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:23:48.761752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.761713 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797"] Nov 03 13:23:48.765595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.765577 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.864797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.864705 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797"] Nov 03 13:23:48.877506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.877472 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtkr\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-kube-api-access-7qtkr\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.877676 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.877522 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846adfab-75a2-4825-af40-05bc343109ae-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.877676 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.877662 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.877820 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.877800 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.877880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.877844 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.877922 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.877897 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.877998 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.877984 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.878090 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.878053 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.878171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.878150 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846adfab-75a2-4825-af40-05bc343109ae-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.882390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.882341 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:48.979203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979166 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.979433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979217 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.979433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979241 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.979433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979291 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.979433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979340 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.979433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979371 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846adfab-75a2-4825-af40-05bc343109ae-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.979433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979424 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtkr\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-kube-api-access-7qtkr\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.979748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979466 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846adfab-75a2-4825-af40-05bc343109ae-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.979748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.979509 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.980302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.980272 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.980832 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.980512 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.981132 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.980800 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.981132 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.981102 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.981274 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.980953 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846adfab-75a2-4825-af40-05bc343109ae-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.982882 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.982860 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:48.983632 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:48.983612 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846adfab-75a2-4825-af40-05bc343109ae-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:49.033913 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.033881 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:49.034089 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.034052 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtkr\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-kube-api-access-7qtkr\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:49.080543 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.080513 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:49.499458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.499433 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797"] Nov 03 13:23:49.501891 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:23:49.501861 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846adfab_75a2_4825_af40_05bc343109ae.slice/crio-bd4f61bb7f1db48eabd1752e712d897ccb3ad88289e52e8c84178f43931be7a8 WatchSource:0}: Error finding container bd4f61bb7f1db48eabd1752e712d897ccb3ad88289e52e8c84178f43931be7a8: Status 404 returned error can't find the container with id bd4f61bb7f1db48eabd1752e712d897ccb3ad88289e52e8c84178f43931be7a8 Nov 03 13:23:49.504250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.504216 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:49.504318 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.504281 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:49.504318 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.504309 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:49.786724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.786617 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" event={"ID":"846adfab-75a2-4825-af40-05bc343109ae","Type":"ContainerStarted","Data":"506ae1324d2a4f11d400c7eea3226d5d24e4af97cdd9facb0c7f43f1a7568586"} Nov 03 13:23:49.786724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.786678 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" event={"ID":"846adfab-75a2-4825-af40-05bc343109ae","Type":"ContainerStarted","Data":"bd4f61bb7f1db48eabd1752e712d897ccb3ad88289e52e8c84178f43931be7a8"} Nov 03 13:23:49.900527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:49.900463 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podStartSLOduration=1.900449006 podStartE2EDuration="1.900449006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:49.897734734 +0000 UTC m=+1163.092138329" watchObservedRunningTime="2026-11-03 13:23:49.900449006 +0000 UTC m=+1163.094852654" Nov 03 13:23:50.081702 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:50.081603 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:50.083163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:50.083133 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:23:50.083292 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:50.083186 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:23:51.080921 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:51.080881 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:23:51.081364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:51.080955 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:23:52.081341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:52.081308 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:23:52.081802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:52.081360 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:23:53.081636 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:53.081595 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:23:53.082189 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:53.081675 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:23:53.964951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:53.964917 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:54.081441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:54.081404 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:23:54.081637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:54.081457 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:23:55.081410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:55.081369 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:23:55.081913 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:55.081430 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:23:56.081036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:56.080994 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:23:56.081237 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:56.081064 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:23:57.086218 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:57.086186 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:57.086712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:57.086596 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:57.087339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:57.087318 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:23:57.879730 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:57.879690 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp"] Nov 03 13:23:57.913598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:57.913566 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:23:59.146951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:23:59.146903 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:04.153103 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.153052 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt"] Nov 03 13:24:04.157143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.157122 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.206092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.206041 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5bb463-openshift-default-dockercfg-jldlg\"" Nov 03 13:24:04.256710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.256675 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt"] Nov 03 13:24:04.268967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.268932 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:04.287288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287254 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.287452 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287317 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.287452 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287412 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.287541 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287458 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszzv\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-kube-api-access-mszzv\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.287541 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287485 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.287637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287608 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.287716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287670 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.287766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287750 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.287812 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.287785 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.388977 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.388942 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389095 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389170 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszzv\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-kube-api-access-mszzv\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389206 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389240 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389282 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389320 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389349 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389556 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389433 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389669 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389641 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389732 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389807 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389853 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.389923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.389913 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.392479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.392457 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.392986 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.392966 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.443177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.443148 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.443329 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.443253 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszzv\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-kube-api-access-mszzv\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.472204 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.472173 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:04.891082 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.891038 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt"] Nov 03 13:24:04.892042 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:24:04.891996 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb36b5d_c3cc_43f6_8ce9_1ead0fb21c15.slice/crio-e6ad493d182c17d8e9648bd4cedc49e06836a515a8d1360984af3717bd1ca6c4 WatchSource:0}: Error finding container e6ad493d182c17d8e9648bd4cedc49e06836a515a8d1360984af3717bd1ca6c4: Status 404 returned error can't find the container with id e6ad493d182c17d8e9648bd4cedc49e06836a515a8d1360984af3717bd1ca6c4 Nov 03 13:24:04.894430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.894398 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:04.894505 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.894477 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:04.894549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:04.894505 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:05.900934 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:05.900889 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" event={"ID":"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15","Type":"ContainerStarted","Data":"a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc"} Nov 03 13:24:05.901415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:05.900980 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" event={"ID":"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15","Type":"ContainerStarted","Data":"e6ad493d182c17d8e9648bd4cedc49e06836a515a8d1360984af3717bd1ca6c4"} Nov 03 13:24:06.106692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:06.106639 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podStartSLOduration=2.1066247750000002 podStartE2EDuration="2.106624775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:06.059562466 +0000 UTC m=+1179.253966060" watchObservedRunningTime="2026-11-03 13:24:06.106624775 +0000 UTC m=+1179.301028369" Nov 03 13:24:06.473424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:06.472479 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:06.474487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:06.474452 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:24:06.474615 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:06.474518 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:24:07.473335 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:07.473289 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:24:07.473731 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:07.473353 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:24:08.472579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:08.472538 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:24:08.472765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:08.472606 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:24:09.473559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:09.473520 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:24:09.474056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:09.473583 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:24:09.496253 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:09.496203 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:10.473132 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:10.473043 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:24:10.473328 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:10.473152 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:24:11.473244 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:11.473211 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:24:11.473722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:11.473270 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:24:12.248949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:12.248907 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:12.477794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:12.477763 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:12.478251 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:12.478185 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:12.479030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:12.479010 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:24:12.907311 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:12.907271 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" podUID="d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" containerName="main" containerID="cri-o://61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9" gracePeriod=45 Nov 03 13:24:13.170405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.170350 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:24:13.219140 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219104 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-dshm\") pod \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " Nov 03 13:24:13.219302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219174 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-home\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-home\") pod \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " Nov 03 13:24:13.219302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219195 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tls-certs\") pod \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " Nov 03 13:24:13.219302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219275 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tmp-dir\") pod \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " Nov 03 13:24:13.219482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219333 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-kube-api-access-xkhzd\" (UniqueName: \"kubernetes.io/projected/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-kube-api-access-xkhzd\") pod \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " Nov 03 13:24:13.219482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219370 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-model-cache\") pod \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\" (UID: \"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3\") " Nov 03 13:24:13.219591 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219476 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-home" pod "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" (UID: "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:13.219654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219598 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tmp-dir" pod "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" (UID: "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:13.219829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219804 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-model-cache" pod "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" (UID: "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:13.219924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219861 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.219924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.219874 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.221649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.221623 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-kube-api-access-xkhzd" pod "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" (UID: "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3"). InnerVolumeSpecName "kube-api-access-xkhzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:13.221755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.221623 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tls-certs" pod "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" (UID: "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:13.221755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.221708 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-dshm" pod "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" (UID: "d2b58384-2e0a-41e5-8a24-6f9fbcb449c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:13.321000 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.320964 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhzd\" (UniqueName: \"kubernetes.io/projected/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-kube-api-access-xkhzd\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.321000 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.320996 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.321000 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.321006 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.321240 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.321015 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.951908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.951872 4924 generic.go:356] "Generic (PLEG): container finished" podID="d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" containerID="61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9" exitCode=0 Nov 03 13:24:13.952363 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.951913 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" event={"ID":"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3","Type":"ContainerDied","Data":"61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9"} Nov 03 13:24:13.952363 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.951930 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" Nov 03 13:24:13.952363 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.951955 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp" event={"ID":"d2b58384-2e0a-41e5-8a24-6f9fbcb449c3","Type":"ContainerDied","Data":"487a80bda37832cb2177d143f465e1185448531d8e4e436ed129bd1781207608"} Nov 03 13:24:13.952363 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.951973 4924 scope.go:122] "RemoveContainer" containerID="61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9" Nov 03 13:24:13.963033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.963011 4924 scope.go:122] "RemoveContainer" containerID="61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9" Nov 03 13:24:13.963480 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:24:13.963451 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9\": container with ID starting with 61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9 not found: ID does not exist" containerID="61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9" Nov 03 13:24:13.963584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:13.963491 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9"} err="failed to get container status \"61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9\": rpc error: code = NotFound desc = could not find container \"61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9\": container with ID starting with 61b66a1e92ebae380e11e17b207fef5904fc0e48dcd3ec537b9b33bc1b0266f9 not found: ID does not exist" Nov 03 13:24:14.119038 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:14.119002 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp"] Nov 03 13:24:14.165214 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:14.165184 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-b-f2a72750-kserve-64d5ff9f87-p6tqp"] Nov 03 13:24:14.589063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:14.589024 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:14.950423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:14.950389 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" path="/var/lib/kubelet/pods/d2b58384-2e0a-41e5-8a24-6f9fbcb449c3/volumes" Nov 03 13:24:16.826675 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:16.826635 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:16.960376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:16.960328 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:17.105891 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:17.105788 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:18.562593 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:24:18.562559 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:24:19.876813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:19.876773 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:20.256564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.256529 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w"] Nov 03 13:24:20.257549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.257522 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" containerName="main" Nov 03 13:24:20.257712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.257697 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" containerName="main" Nov 03 13:24:20.258005 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.257985 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b58384-2e0a-41e5-8a24-6f9fbcb449c3" containerName="main" Nov 03 13:24:20.279446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.279412 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.313327 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.313297 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.313540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.313523 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.313649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.313638 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.313733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.313723 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.313828 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.313815 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.313899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.313889 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss26f\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-kube-api-access-ss26f\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.313974 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.313965 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.314053 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.314044 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.314313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.314290 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.417600 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.417687 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.417722 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss26f\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-kube-api-access-ss26f\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.417752 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.417795 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.417837 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.417958 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.418010 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.418053 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.418897 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w"] Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.419894 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.420578 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.420838 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.421160 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.421896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.421411 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.442089 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.440350 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.442089 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.440842 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.515491 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.515424 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.516334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.516306 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss26f\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-kube-api-access-ss26f\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:20.602639 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:20.602605 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:21.038611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:21.038577 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w"] Nov 03 13:24:21.039487 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:24:21.039458 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b974cad_c05f_4cd4_a8e5_a73a4acfd506.slice/crio-8f3188e3260746b14e896a68cd4b0856671d81ab829ec88e477b4a8358918f75 WatchSource:0}: Error finding container 8f3188e3260746b14e896a68cd4b0856671d81ab829ec88e477b4a8358918f75: Status 404 returned error can't find the container with id 8f3188e3260746b14e896a68cd4b0856671d81ab829ec88e477b4a8358918f75 Nov 03 13:24:21.042128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:21.042092 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:21.042214 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:21.042158 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:21.042214 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:21.042185 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:22.021354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:22.021313 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" event={"ID":"1b974cad-c05f-4cd4-a8e5-a73a4acfd506","Type":"ContainerStarted","Data":"faf301a4c1b548744caa5f744497869fd9074b5820d446353ee728a34de25f90"} Nov 03 13:24:22.021545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:22.021377 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" event={"ID":"1b974cad-c05f-4cd4-a8e5-a73a4acfd506","Type":"ContainerStarted","Data":"8f3188e3260746b14e896a68cd4b0856671d81ab829ec88e477b4a8358918f75"} Nov 03 13:24:22.182821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:22.182765 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podStartSLOduration=2.182746337 podStartE2EDuration="2.182746337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:22.178176632 +0000 UTC m=+1195.372580225" watchObservedRunningTime="2026-11-03 13:24:22.182746337 +0000 UTC m=+1195.377149930" Nov 03 13:24:22.603153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:22.603114 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:22.604420 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:22.604398 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:24:22.604486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:22.604457 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:24:23.345898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:23.345863 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:23.603733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:23.603652 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:24:23.603733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:23.603706 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:24:24.603782 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:24.603745 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:24:24.604301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:24.603803 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:24:25.247613 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:25.247577 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:25.603797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:25.603707 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:24:25.603797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:25.603777 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:24:26.603430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:26.603395 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:24:26.603630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:26.603455 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:24:27.105592 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:27.105556 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:24:27.107769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:27.107745 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:24:27.603978 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:27.603935 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:24:27.604177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:27.604021 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:24:28.607982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:28.607946 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:28.608385 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:28.608299 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:28.608937 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:28.608920 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:24:30.550440 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:30.550399 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:35.826481 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:35.826431 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:41.215635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:41.215499 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:43.842094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:43.835913 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:24:43.842094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:43.835990 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:24:46.468966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:46.468927 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:48.559051 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:24:48.559009 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:24:51.669604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:51.669566 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:24:56.781092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:24:56.781045 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:00.089302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:00.089261 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:02.210543 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:02.210493 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:05.664849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.664810 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:05.686975 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.686941 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc"] Nov 03 13:25:05.692636 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.692616 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.803177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.803138 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.803503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.803479 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.803665 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.803648 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.803803 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.803788 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtc6\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-kube-api-access-hmtc6\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.804167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.804142 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ef34d-0799-40fe-b18c-2681d8528771-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.804327 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.804311 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ef34d-0799-40fe-b18c-2681d8528771-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.804446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.804432 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.804586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.804573 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.804704 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.804691 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.807888 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.807864 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc"] Nov 03 13:25:05.906289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906255 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ef34d-0799-40fe-b18c-2681d8528771-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906294 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ef34d-0799-40fe-b18c-2681d8528771-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906317 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906360 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906392 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906451 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906481 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906529 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906838 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906575 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtc6\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-kube-api-access-hmtc6\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906922 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906899 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906994 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906921 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.906994 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.906962 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.907217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.907191 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.907878 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.907852 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ef34d-0799-40fe-b18c-2681d8528771-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.909420 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.909392 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ef34d-0799-40fe-b18c-2681d8528771-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.910316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.910288 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.989141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.980317 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtc6\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-kube-api-access-hmtc6\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:05.989141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:05.982267 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:06.012012 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:06.011973 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:06.449354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:06.449327 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc"] Nov 03 13:25:06.450798 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:25:06.450771 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17ef34d_0799_40fe_b18c_2681d8528771.slice/crio-6802df3f3c6e2bbb687ded5bf482bfa196f62db2fa2dada96c3b142f48427f41 WatchSource:0}: Error finding container 6802df3f3c6e2bbb687ded5bf482bfa196f62db2fa2dada96c3b142f48427f41: Status 404 returned error can't find the container with id 6802df3f3c6e2bbb687ded5bf482bfa196f62db2fa2dada96c3b142f48427f41 Nov 03 13:25:06.453187 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:06.453155 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:06.453275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:06.453219 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:06.453275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:06.453248 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:06.545695 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:06.545652 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" event={"ID":"e17ef34d-0799-40fe-b18c-2681d8528771","Type":"ContainerStarted","Data":"8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2"} Nov 03 13:25:06.545847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:06.545713 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" event={"ID":"e17ef34d-0799-40fe-b18c-2681d8528771","Type":"ContainerStarted","Data":"6802df3f3c6e2bbb687ded5bf482bfa196f62db2fa2dada96c3b142f48427f41"} Nov 03 13:25:06.661143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:06.661064 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podStartSLOduration=1.661044917 podStartE2EDuration="1.661044917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:06.656336652 +0000 UTC m=+1239.850740249" watchObservedRunningTime="2026-11-03 13:25:06.661044917 +0000 UTC m=+1239.855448515" Nov 03 13:25:07.012944 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:07.012905 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:07.014621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:07.014591 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 13:25:07.014742 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:07.014649 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 13:25:07.544093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:07.526161 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:08.012723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:08.012689 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 13:25:08.012918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:08.012751 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 13:25:09.013093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:09.013036 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 13:25:09.013483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:09.013110 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 13:25:10.012695 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:10.012659 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 13:25:10.012863 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:10.012715 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 13:25:11.013283 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:11.013243 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 13:25:11.013690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:11.013310 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 13:25:12.013495 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:12.013459 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 13:25:12.013897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:12.013523 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 13:25:12.853472 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:12.853435 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:13.012796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:13.012759 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 13:25:13.012998 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:13.012812 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 13:25:14.016865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:14.016829 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:14.017300 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:14.017122 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:14.017693 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:14.017679 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:25:17.961297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:17.961258 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:18.559468 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:25:18.559433 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:25:21.437572 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:21.437519 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:23.679685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:23.679648 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:23.941587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:23.941505 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:24.210571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.210492 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764"] Nov 03 13:25:24.214747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.214721 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.264415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.264356 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:25:24.293212 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.293172 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.293382 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.293232 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.293382 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.293361 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nhr\" (UniqueName: \"kubernetes.io/projected/69ee6910-4a13-4a35-af0f-541c0fd73659-kube-api-access-d4nhr\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.310861 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.310825 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:25:24.314265 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.314239 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764"] Nov 03 13:25:24.395206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.395162 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.395408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.395261 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.395408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.395349 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nhr\" (UniqueName: \"kubernetes.io/projected/69ee6910-4a13-4a35-af0f-541c0fd73659-kube-api-access-d4nhr\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.395920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.395897 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.397752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.397726 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.453211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.453180 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nhr\" (UniqueName: \"kubernetes.io/projected/69ee6910-4a13-4a35-af0f-541c0fd73659-kube-api-access-d4nhr\") pod \"maas-api-revoke-sub-ee1853b0-g2764\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.529213 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.529139 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:24.898291 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.898254 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764"] Nov 03 13:25:24.899255 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:25:24.899227 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ee6910_4a13_4a35_af0f_541c0fd73659.slice/crio-aa5837906a7d748caa028bdaffb6b3184fe004d533eeabf4f397ef6ad34d5d0f WatchSource:0}: Error finding container aa5837906a7d748caa028bdaffb6b3184fe004d533eeabf4f397ef6ad34d5d0f: Status 404 returned error can't find the container with id aa5837906a7d748caa028bdaffb6b3184fe004d533eeabf4f397ef6ad34d5d0f Nov 03 13:25:24.901519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:24.901502 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:25:25.721484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:25.721450 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" event={"ID":"69ee6910-4a13-4a35-af0f-541c0fd73659","Type":"ContainerStarted","Data":"aa5837906a7d748caa028bdaffb6b3184fe004d533eeabf4f397ef6ad34d5d0f"} Nov 03 13:25:29.158768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:29.158722 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:29.759867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:29.759822 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" event={"ID":"69ee6910-4a13-4a35-af0f-541c0fd73659","Type":"ContainerStarted","Data":"9ddd56b16e275433dbbeb46620441202eeddd422b473aba6582ab95f0390ef34"} Nov 03 13:25:29.874551 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:29.874485 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" podStartSLOduration=1.8833514569999998 podStartE2EDuration="5.874465346s" totalImagesPullingTime="3.991113889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:29.868694703 +0000 UTC m=+1263.063098299" watchObservedRunningTime="2026-11-03 13:25:29.874465346 +0000 UTC m=+1263.068868937" Nov 03 13:25:30.768116 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:30.768059 4924 generic.go:356] "Generic (PLEG): container finished" podID="69ee6910-4a13-4a35-af0f-541c0fd73659" containerID="9ddd56b16e275433dbbeb46620441202eeddd422b473aba6582ab95f0390ef34" exitCode=0 Nov 03 13:25:30.768508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:30.768143 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" event={"ID":"69ee6910-4a13-4a35-af0f-541c0fd73659","Type":"ContainerDied","Data":"9ddd56b16e275433dbbeb46620441202eeddd422b473aba6582ab95f0390ef34"} Nov 03 13:25:31.934714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:31.934686 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:32.084093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.083988 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-openshift-ca\") pod \"69ee6910-4a13-4a35-af0f-541c0fd73659\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " Nov 03 13:25:32.084268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.084119 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-xks-ca\") pod \"69ee6910-4a13-4a35-af0f-541c0fd73659\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " Nov 03 13:25:32.084268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.084194 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ee6910-4a13-4a35-af0f-541c0fd73659-kube-api-access-d4nhr\" (UniqueName: \"kubernetes.io/projected/69ee6910-4a13-4a35-af0f-541c0fd73659-kube-api-access-d4nhr\") pod \"69ee6910-4a13-4a35-af0f-541c0fd73659\" (UID: \"69ee6910-4a13-4a35-af0f-541c0fd73659\") " Nov 03 13:25:32.084454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.084427 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-openshift-ca" pod "69ee6910-4a13-4a35-af0f-541c0fd73659" (UID: "69ee6910-4a13-4a35-af0f-541c0fd73659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:32.084682 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.084664 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:32.086305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.086279 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ee6910-4a13-4a35-af0f-541c0fd73659-kube-api-access-d4nhr" pod "69ee6910-4a13-4a35-af0f-541c0fd73659" (UID: "69ee6910-4a13-4a35-af0f-541c0fd73659"). InnerVolumeSpecName "kube-api-access-d4nhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:32.086666 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.086648 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-xks-ca" pod "69ee6910-4a13-4a35-af0f-541c0fd73659" (UID: "69ee6910-4a13-4a35-af0f-541c0fd73659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:32.185630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.185596 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ee6910-4a13-4a35-af0f-541c0fd73659-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:32.185630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.185628 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4nhr\" (UniqueName: \"kubernetes.io/projected/69ee6910-4a13-4a35-af0f-541c0fd73659-kube-api-access-d4nhr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:32.783004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.782967 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" Nov 03 13:25:32.783004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.782986 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764" event={"ID":"69ee6910-4a13-4a35-af0f-541c0fd73659","Type":"ContainerDied","Data":"aa5837906a7d748caa028bdaffb6b3184fe004d533eeabf4f397ef6ad34d5d0f"} Nov 03 13:25:32.783321 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:32.783034 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5837906a7d748caa028bdaffb6b3184fe004d533eeabf4f397ef6ad34d5d0f" Nov 03 13:25:34.399094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.398874 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:34.684096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.683567 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk"] Nov 03 13:25:34.692094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.684381 4924 cpu_manager.go:414] "containerMap: removing container" podUID="69ee6910-4a13-4a35-af0f-541c0fd73659" containerName="revoke-keys" Nov 03 13:25:34.692094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.684404 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ee6910-4a13-4a35-af0f-541c0fd73659" containerName="revoke-keys" Nov 03 13:25:34.692094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.684523 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ee6910-4a13-4a35-af0f-541c0fd73659" containerName="revoke-keys" Nov 03 13:25:34.729095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.721394 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.811339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.811292 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.811756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.811734 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733a046-e91a-4430-9d6c-5756d05b91f1-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.811945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.811928 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.812171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.812144 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.812312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.812294 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.812436 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.812406 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.812567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.812551 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.812696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.812669 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.812821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.812805 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrz65\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-kube-api-access-rrz65\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.813221 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.813181 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 13:25:34.886232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.886134 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk"] Nov 03 13:25:34.929898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.929769 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.929898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.929826 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.929898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.929863 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.929898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.929892 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.930286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.929932 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.930286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.929960 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrz65\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-kube-api-access-rrz65\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.930286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.930099 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.930286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.930144 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733a046-e91a-4430-9d6c-5756d05b91f1-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.930286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.930204 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.961185 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.957969 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.975164 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.972813 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.975758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.975729 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.997945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.976218 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733a046-e91a-4430-9d6c-5756d05b91f1-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.999205 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.997962 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:34.999557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:34.999519 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:35.002056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.002031 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:35.016759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.008728 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrz65\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-kube-api-access-rrz65\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:35.016759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.009745 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:35.058383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.058352 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:35.492655 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.492615 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk"] Nov 03 13:25:35.528716 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:25:35.528675 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6733a046_e91a_4430_9d6c_5756d05b91f1.slice/crio-08ee2db3cd524e8ed6de02115d86e2998ea4192ca7e8c9b077b1bfc8271623bc WatchSource:0}: Error finding container 08ee2db3cd524e8ed6de02115d86e2998ea4192ca7e8c9b077b1bfc8271623bc: Status 404 returned error can't find the container with id 08ee2db3cd524e8ed6de02115d86e2998ea4192ca7e8c9b077b1bfc8271623bc Nov 03 13:25:35.530882 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.530842 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:35.530986 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.530934 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:35.530986 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.530978 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:35.876151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.876049 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" event={"ID":"6733a046-e91a-4430-9d6c-5756d05b91f1","Type":"ContainerStarted","Data":"9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591"} Nov 03 13:25:35.876151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.876133 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" event={"ID":"6733a046-e91a-4430-9d6c-5756d05b91f1","Type":"ContainerStarted","Data":"08ee2db3cd524e8ed6de02115d86e2998ea4192ca7e8c9b077b1bfc8271623bc"} Nov 03 13:25:35.984041 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.983999 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:35.991250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:35.991200 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podStartSLOduration=1.9911810490000001 podStartE2EDuration="1.991181049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:35.988685357 +0000 UTC m=+1269.183088951" watchObservedRunningTime="2026-11-03 13:25:35.991181049 +0000 UTC m=+1269.185584645" Nov 03 13:25:36.059429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.059334 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:36.067146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.065202 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:25:36.067146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.065263 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:25:36.161525 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.161488 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:36.517465 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.517377 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:36.738341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.738304 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7"] Nov 03 13:25:36.758593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.758558 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.830127 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.816024 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5295ab29-openshift-default-dockercfg-h5sdt\"" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917004 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff8111-1807-47fd-a147-af28bae5cf12-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917101 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917126 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917163 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917232 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff8111-1807-47fd-a147-af28bae5cf12-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917276 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917321 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917381 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.917444 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smdw\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-kube-api-access-8smdw\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:36.923091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.921745 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:36.966643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:36.930467 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7"] Nov 03 13:25:37.025051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.025018 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smdw\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-kube-api-access-8smdw\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.025333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.025313 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff8111-1807-47fd-a147-af28bae5cf12-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.025472 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.025458 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.025566 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.025552 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.025666 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.025653 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.025809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.025797 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff8111-1807-47fd-a147-af28bae5cf12-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.025924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.025912 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.026030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.026016 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.029280 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.029252 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.051182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.038717 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.080114 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.052177 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff8111-1807-47fd-a147-af28bae5cf12-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.085177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.085141 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:25:37.085405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.085375 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:25:37.085548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.053846 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.089602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.053633 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.090250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.090201 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.155177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.155133 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.200105 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.175625 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff8111-1807-47fd-a147-af28bae5cf12-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.201784 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.201752 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:37.204873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.204749 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.204873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.204826 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smdw\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-kube-api-access-8smdw\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.424156 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.424117 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:37.579093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.578159 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm"] Nov 03 13:25:37.585294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.585124 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.647384 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.647351 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"distinct-2-w0-42c707-kserve-self-signed-certs\"" Nov 03 13:25:37.663272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.652471 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm"] Nov 03 13:25:37.678048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.677887 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:37.767202 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.767155 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tls-certs\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.767423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.767234 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-model-cache\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.767423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.767272 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxjkh\" (UniqueName: \"kubernetes.io/projected/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-kube-api-access-lxjkh\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.767423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.767408 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-home\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.767602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.767478 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tmp-dir\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.767602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.767569 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-dshm\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.868601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.868560 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-home\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.868817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.868642 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tmp-dir\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.868817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.868721 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-dshm\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.868949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.868840 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tls-certs\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.868949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.868886 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-model-cache\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.868949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.868913 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxjkh\" (UniqueName: \"kubernetes.io/projected/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-kube-api-access-lxjkh\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.869713 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.869687 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-home\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.869969 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.869947 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tmp-dir\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.870242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.870220 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-model-cache\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.872495 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.872470 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tls-certs\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.872977 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.872956 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-dshm\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.878214 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.878188 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7"] Nov 03 13:25:37.879948 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:25:37.879923 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dff8111_1807_47fd_a147_af28bae5cf12.slice/crio-8b0f84aaa8a74893fafb20e39ec5f9c79b5bacc3c260f7dc0b856cfeaf216e5d WatchSource:0}: Error finding container 8b0f84aaa8a74893fafb20e39ec5f9c79b5bacc3c260f7dc0b856cfeaf216e5d: Status 404 returned error can't find the container with id 8b0f84aaa8a74893fafb20e39ec5f9c79b5bacc3c260f7dc0b856cfeaf216e5d Nov 03 13:25:37.882413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.882379 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:37.882518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.882461 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:37.882518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.882510 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:37.959931 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.954333 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxjkh\" (UniqueName: \"kubernetes.io/projected/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-kube-api-access-lxjkh\") pod \"distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:37.975118 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:37.974998 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" event={"ID":"7dff8111-1807-47fd-a147-af28bae5cf12","Type":"ContainerStarted","Data":"8b0f84aaa8a74893fafb20e39ec5f9c79b5bacc3c260f7dc0b856cfeaf216e5d"} Nov 03 13:25:38.077102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:38.065279 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:25:38.077102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:38.065348 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:25:38.201283 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:38.201245 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:38.633609 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:25:38.632219 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd193bf67_8de9_4eb6_b6cc_51d51960cd2b.slice/crio-c00ca383be6589dbec6898db6d427103e83a6f0aef06fa64cc2aba61e03b7acf WatchSource:0}: Error finding container c00ca383be6589dbec6898db6d427103e83a6f0aef06fa64cc2aba61e03b7acf: Status 404 returned error can't find the container with id c00ca383be6589dbec6898db6d427103e83a6f0aef06fa64cc2aba61e03b7acf Nov 03 13:25:38.633609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:38.633378 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm"] Nov 03 13:25:38.983544 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:38.983482 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" event={"ID":"7dff8111-1807-47fd-a147-af28bae5cf12","Type":"ContainerStarted","Data":"2625bfbe7ec29b82a2841f10c40e68ede4fbaca19168abff1a364b4edfe82e6a"} Nov 03 13:25:38.985782 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:38.985756 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" event={"ID":"d193bf67-8de9-4eb6-b6cc-51d51960cd2b","Type":"ContainerStarted","Data":"283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa"} Nov 03 13:25:38.985906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:38.985793 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" event={"ID":"d193bf67-8de9-4eb6-b6cc-51d51960cd2b","Type":"ContainerStarted","Data":"c00ca383be6589dbec6898db6d427103e83a6f0aef06fa64cc2aba61e03b7acf"} Nov 03 13:25:39.059577 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.059535 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:25:39.059734 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.059592 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:25:39.115914 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.115858 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" podStartSLOduration=3.115844246 podStartE2EDuration="3.115844246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:39.113461788 +0000 UTC m=+1272.307865385" watchObservedRunningTime="2026-11-03 13:25:39.115844246 +0000 UTC m=+1272.310247840" Nov 03 13:25:39.201985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.201949 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:39.222229 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.222178 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" podStartSLOduration=2.222163695 podStartE2EDuration="2.222163695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:39.221182158 +0000 UTC m=+1272.415585752" watchObservedRunningTime="2026-11-03 13:25:39.222163695 +0000 UTC m=+1272.416567330" Nov 03 13:25:39.425235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.425203 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:39.441645 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.441370 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:39.993424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.993384 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:39.994200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:39.994176 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:25:40.059596 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:40.059560 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:25:40.059770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:40.059624 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:25:41.059137 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:41.059095 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:25:41.059540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:41.059154 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:25:41.925233 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:41.925196 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:42.058836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:42.058797 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:25:42.059003 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:42.058853 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:25:42.166924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:42.166882 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:42.440889 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:42.440850 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:42.724519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:42.724436 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:43.063786 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:43.063698 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:43.064134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:43.064112 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:43.064815 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:43.064799 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:25:45.528016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:45.527978 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:45.708637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:45.708607 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:47.859682 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:47.859616 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:48.202657 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:48.202618 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:48.215521 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:48.215490 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:48.577515 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:25:48.577430 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:25:49.080418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:49.080382 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 13:25:49.238123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:49.238039 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:49.393158 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:49.393033 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:50.048484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.048441 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2"] Nov 03 13:25:50.069478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.069444 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.151843 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.151970 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.152000 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68f6071-52f8-45ac-9aac-6a917842cdda-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.152111 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.152173 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.152202 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.152229 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.152274 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.164096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.152305 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jbfs\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-kube-api-access-5jbfs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.198097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.166551 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2"] Nov 03 13:25:50.256084 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.256027 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.257563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.256830 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68f6071-52f8-45ac-9aac-6a917842cdda-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.257814 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.257794 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.258292 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.257528 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68f6071-52f8-45ac-9aac-6a917842cdda-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.258429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.256756 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.258558 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.258186 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.258692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.258675 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.259144 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.259124 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.259618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.259599 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.260246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.259535 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.260755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.260733 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.260899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.260878 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.261032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.261014 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jbfs\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-kube-api-access-5jbfs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.296046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.296008 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.297545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.297516 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.375143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.375026 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.452985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.452948 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.474022 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.473985 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jbfs\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-kube-api-access-5jbfs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:50.547602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:50.547296 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:51.705651 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:51.703857 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2"] Nov 03 13:25:51.802027 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:51.801968 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:51.802333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:51.802299 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:51.802507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:51.802484 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:52.461521 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:52.461480 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" event={"ID":"c68f6071-52f8-45ac-9aac-6a917842cdda","Type":"ContainerStarted","Data":"40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff"} Nov 03 13:25:52.461779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:52.461758 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" event={"ID":"c68f6071-52f8-45ac-9aac-6a917842cdda","Type":"ContainerStarted","Data":"1715c16f74e9659cb786000ba1bb6755e5434717579ffb04e6d79e20738ae7cf"} Nov 03 13:25:52.548559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:52.548467 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:52.624033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:52.623968 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" podStartSLOduration=3.623947453 podStartE2EDuration="3.623947453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:52.620172199 +0000 UTC m=+1285.814575793" watchObservedRunningTime="2026-11-03 13:25:52.623947453 +0000 UTC m=+1285.818351050" Nov 03 13:25:53.550135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:53.549757 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:53.550135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:53.549824 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 13:25:54.464238 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:54.464200 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:54.553105 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:54.551459 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:54.553105 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:54.551523 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 13:25:55.551496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:55.551454 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:55.551806 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:55.551770 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 13:25:56.554092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:56.553463 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:56.554092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:56.553529 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 13:25:57.554511 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:57.554466 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:57.555033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:57.554535 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 13:25:58.548298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:58.548261 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:58.548495 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:58.548318 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 13:25:59.717137 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:59.717060 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:25:59.912045 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:59.912010 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:59.912939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:59.912909 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:25:59.913804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:25:59.913774 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:26:00.131772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:00.131692 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:00.449922 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:00.449886 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:02.324748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:02.324709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:02.443388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:02.443357 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:04.761163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:04.761125 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:07.465864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:07.465773 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:07.618366 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:07.618333 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:07.757385 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:07.757298 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:08.857418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:08.857376 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:08.991329 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:08.991286 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:09.121720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:09.121634 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:10.014825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:10.014789 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:11.473629 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.473592 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l"] Nov 03 13:26:11.477971 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.477948 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.579183 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.579142 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l"] Nov 03 13:26:11.644389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644354 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.644600 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644396 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ptr\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-kube-api-access-w2ptr\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.644600 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644503 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.644600 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644544 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.644812 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644676 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.644812 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644733 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.644936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644865 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.644936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644911 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.645055 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.644961 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb9f30-a784-4180-aa25-64a3e8e79381-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.747035 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.746948 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.747199 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747095 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.747199 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747148 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.747199 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747182 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.747358 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747230 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb9f30-a784-4180-aa25-64a3e8e79381-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.747358 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747314 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.747916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747883 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ptr\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-kube-api-access-w2ptr\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.748036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747375 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.748036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747595 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.748036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747473 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.748036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747671 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.748036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.747996 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.748353 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.748044 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.748353 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.748040 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb9f30-a784-4180-aa25-64a3e8e79381-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.749728 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.749703 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.750361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.750343 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.801768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.801740 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:11.801889 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:11.801860 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ptr\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-kube-api-access-w2ptr\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:12.091337 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.091249 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:12.154117 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.152443 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd"] Nov 03 13:26:12.154117 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.152697 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" containerID="cri-o://b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea" gracePeriod=30 Nov 03 13:26:12.534104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.534060 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l"] Nov 03 13:26:12.534779 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:26:12.534754 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bb9f30_a784_4180_aa25_64a3e8e79381.slice/crio-4b8bcde74a75ea798e808fbe93068a580c7896376498046d13462215d09f8547 WatchSource:0}: Error finding container 4b8bcde74a75ea798e808fbe93068a580c7896376498046d13462215d09f8547: Status 404 returned error can't find the container with id 4b8bcde74a75ea798e808fbe93068a580c7896376498046d13462215d09f8547 Nov 03 13:26:12.537242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.537206 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:12.537328 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.537273 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:12.537328 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.537304 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:12.607277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.607245 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" event={"ID":"e1bb9f30-a784-4180-aa25-64a3e8e79381","Type":"ContainerStarted","Data":"aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77"} Nov 03 13:26:12.607416 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.607300 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" event={"ID":"e1bb9f30-a784-4180-aa25-64a3e8e79381","Type":"ContainerStarted","Data":"4b8bcde74a75ea798e808fbe93068a580c7896376498046d13462215d09f8547"} Nov 03 13:26:12.722446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:12.722387 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podStartSLOduration=1.722370221 podStartE2EDuration="1.722370221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:12.719308171 +0000 UTC m=+1305.913711766" watchObservedRunningTime="2026-11-03 13:26:12.722370221 +0000 UTC m=+1305.916773814" Nov 03 13:26:13.098097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:13.096539 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:13.116283 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:13.116238 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:26:13.116557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:13.116504 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 13:26:14.096096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:14.095441 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:26:14.096096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:14.095504 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 13:26:14.673103 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:14.672671 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:26:14.673103 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:14.672743 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.839151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:14.815196 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:15.094148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.093469 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:26:15.094148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.093533 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 13:26:15.135097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.119204 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:15.490116 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.488355 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr"] Nov 03 13:26:15.492518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.492490 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.518448 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.518415 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr"] Nov 03 13:26:15.521563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.521507 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-metrics-certs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.521818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.521792 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.522036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.522012 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tshs\" (UniqueName: \"kubernetes.io/projected/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-kube-api-access-7tshs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.522169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.522139 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-openshift-service-ca\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.564025 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.563983 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5295ab29-serving-cert\"" Nov 03 13:26:15.623432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.623394 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tshs\" (UniqueName: \"kubernetes.io/projected/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-kube-api-access-7tshs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.623684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.623505 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-openshift-service-ca\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.623684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.623582 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-metrics-certs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.623684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.623613 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.623889 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:26:15.623826 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-serving-cert: secret "maas-api-e2e-mt-5295ab29-serving-cert" not found Nov 03 13:26:15.623945 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:26:15.623887 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls podName:ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624 nodeName:}" failed. No retries permitted until 2026-11-03 13:26:16.123867429 +0000 UTC m=+1309.318271014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls") pod "maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" (UID: "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624") : secret "maas-api-e2e-mt-5295ab29-serving-cert" not found Nov 03 13:26:15.624556 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.624525 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-openshift-service-ca\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.656123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.649693 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-metrics-certs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.680618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.680577 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tshs\" (UniqueName: \"kubernetes.io/projected/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-kube-api-access-7tshs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:15.697152 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.697121 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:15.760265 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.760118 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:26:15.760265 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.760187 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:26:15.958295 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:15.958257 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:16.086841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:16.086739 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:16.099459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:16.099422 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:26:16.099716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:16.099489 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 13:26:16.135095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:16.134808 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:16.190707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:16.190668 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:16.413000 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:16.412965 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:16.815176 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:16.815147 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr"] Nov 03 13:26:16.815709 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:26:16.815677 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2fe15a_4cf3_48c4_a8e4_46ca1f2f6624.slice/crio-7a0e366f1c4ce65bdf15812d2844b086104ed9cfe867d95e3d8ea59501a07fea WatchSource:0}: Error finding container 7a0e366f1c4ce65bdf15812d2844b086104ed9cfe867d95e3d8ea59501a07fea: Status 404 returned error can't find the container with id 7a0e366f1c4ce65bdf15812d2844b086104ed9cfe867d95e3d8ea59501a07fea Nov 03 13:26:17.093769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.091837 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:26:17.093769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.091905 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 13:26:17.479724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.479695 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:26:17.567895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.567857 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-envoy\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.567909 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590429ba-2e0e-44c7-9635-4b60db71b247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590429ba-2e0e-44c7-9635-4b60db71b247-istiod-ca-cert\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568005 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/590429ba-2e0e-44c7-9635-4b60db71b247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590429ba-2e0e-44c7-9635-4b60db71b247-istio-podinfo\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568054 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-socket\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568122 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-data\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568168 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-kube-api-access-4fv8d\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-kube-api-access-4fv8d\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568293 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-credential-socket\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568351 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-istio-token\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-istio-token\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568380 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-certs\") pod \"590429ba-2e0e-44c7-9635-4b60db71b247\" (UID: \"590429ba-2e0e-44c7-9635-4b60db71b247\") " Nov 03 13:26:17.568474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568415 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-data" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:17.568474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568432 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-socket" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:17.568474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568464 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590429ba-2e0e-44c7-9635-4b60db71b247-istiod-ca-cert" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:26:17.568681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568610 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-credential-socket" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:17.568822 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568802 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.568887 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568828 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.568887 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568830 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-certs" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:17.568887 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568843 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.568887 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.568858 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590429ba-2e0e-44c7-9635-4b60db71b247-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.570310 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.570281 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/590429ba-2e0e-44c7-9635-4b60db71b247-istio-podinfo" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:26:17.570407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.570393 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-kube-api-access-4fv8d" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "kube-api-access-4fv8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:17.570619 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.570601 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-envoy" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:17.570675 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.570630 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-istio-token" pod "590429ba-2e0e-44c7-9635-4b60db71b247" (UID: "590429ba-2e0e-44c7-9635-4b60db71b247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:17.670096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.670032 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590429ba-2e0e-44c7-9635-4b60db71b247-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.670266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.670104 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fv8d\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-kube-api-access-4fv8d\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.670266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.670125 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590429ba-2e0e-44c7-9635-4b60db71b247-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.670266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.670139 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.670266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.670153 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590429ba-2e0e-44c7-9635-4b60db71b247-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:17.687878 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.687839 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" event={"ID":"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624","Type":"ContainerStarted","Data":"7a0e366f1c4ce65bdf15812d2844b086104ed9cfe867d95e3d8ea59501a07fea"} Nov 03 13:26:17.689371 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.689343 4924 generic.go:356] "Generic (PLEG): container finished" podID="590429ba-2e0e-44c7-9635-4b60db71b247" containerID="b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea" exitCode=0 Nov 03 13:26:17.689492 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.689390 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" Nov 03 13:26:17.689492 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.689403 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" event={"ID":"590429ba-2e0e-44c7-9635-4b60db71b247","Type":"ContainerDied","Data":"b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea"} Nov 03 13:26:17.689492 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.689460 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd" event={"ID":"590429ba-2e0e-44c7-9635-4b60db71b247","Type":"ContainerDied","Data":"0e4e5d32806e35584c1dacc0e5abe08bc7f21e9788c5fea6831642908bc63ea6"} Nov 03 13:26:17.689492 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.689483 4924 scope.go:122] "RemoveContainer" containerID="b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea" Nov 03 13:26:17.699535 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.699512 4924 scope.go:122] "RemoveContainer" containerID="b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea" Nov 03 13:26:17.699825 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:26:17.699805 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea\": container with ID starting with b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea not found: ID does not exist" containerID="b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea" Nov 03 13:26:17.699898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.699833 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea"} err="failed to get container status \"b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea\": rpc error: code = NotFound desc = could not find container \"b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea\": container with ID starting with b2f9df7e472da170ddcffc990c1b01ba2d3a9ed320e71c65704d20322aa38eea not found: ID does not exist" Nov 03 13:26:17.873967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.866153 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd"] Nov 03 13:26:17.874191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:17.874050 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-v84sd"] Nov 03 13:26:18.092167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:18.092134 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:26:18.092316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:18.092191 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 13:26:18.578413 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:26:18.578377 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:26:18.698999 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:18.698958 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" event={"ID":"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624","Type":"ContainerStarted","Data":"2f4c47279ab8849108fc9013da3b416428092273cb456052ad8d61be78a73761"} Nov 03 13:26:18.699200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:18.699046 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:18.820485 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:18.820421 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" podStartSLOduration=2.749647843 podStartE2EDuration="3.820400726s" totalImagesPullingTime="1.070752883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:18.816160522 +0000 UTC m=+1312.010564115" watchObservedRunningTime="2026-11-03 13:26:18.820400726 +0000 UTC m=+1312.014804323" Nov 03 13:26:18.986975 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:18.986943 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" path="/var/lib/kubelet/pods/590429ba-2e0e-44c7-9635-4b60db71b247/volumes" Nov 03 13:26:19.092543 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:19.092503 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:26:19.092727 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:19.092566 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 13:26:20.092015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:20.091971 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:26:20.092444 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:20.092039 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 13:26:21.097106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:21.097060 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:21.097683 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:21.097552 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:21.098309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:21.098291 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:26:21.485396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:21.485367 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:21.769036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:21.768949 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:24.132638 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:24.132591 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:24.712896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:24.712870 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:26:26.670240 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:26.670206 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:26.813423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:26.813392 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:27.459645 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:27.459556 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:31.874368 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:31.874333 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:32.872168 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:32.872119 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:37.080948 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:37.080910 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:41.913737 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:41.913703 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:42.229174 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:42.220505 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:42.466095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:42.465696 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:42.920203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:42.920161 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:42.942213 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:42.942162 4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Nov 03 13:26:43.102203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:43.102157 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" podUID="077a2a59-e1a7-4f11-af5d-4373cc6643ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:43.121252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:43.121206 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:43.392153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:43.389191 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:43.900871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:43.900827 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:26:43.902112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:43.902049 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:43.923187 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:43.923139 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:43.934983 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:43.934950 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:26:44.008816 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:44.007829 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 13:26:47.625595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:47.625558 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:48.560519 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:26:48.560473 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:26:49.993831 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:49.993752 4924 generic.go:356] "Generic (PLEG): container finished" podID="7dff8111-1807-47fd-a147-af28bae5cf12" containerID="2625bfbe7ec29b82a2841f10c40e68ede4fbaca19168abff1a364b4edfe82e6a" exitCode=0 Nov 03 13:26:49.993831 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:49.993789 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" event={"ID":"7dff8111-1807-47fd-a147-af28bae5cf12","Type":"ContainerDied","Data":"2625bfbe7ec29b82a2841f10c40e68ede4fbaca19168abff1a364b4edfe82e6a"} Nov 03 13:26:49.994302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:49.994202 4924 scope.go:122] "RemoveContainer" containerID="2625bfbe7ec29b82a2841f10c40e68ede4fbaca19168abff1a364b4edfe82e6a" Nov 03 13:26:49.995575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:49.995543 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:49.995640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:49.995621 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:49.995681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:49.995668 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:50.560496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:50.560462 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:51.002155 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:51.002111 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" event={"ID":"7dff8111-1807-47fd-a147-af28bae5cf12","Type":"ContainerStarted","Data":"90b97bff2bbca93cb53c5c1b49a52ba5ac887d3986cd407b9d23de99d8379888"} Nov 03 13:26:51.427092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:51.424295 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:26:51.427092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:51.425621 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:26:51.427092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:51.425702 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 13:26:51.889135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:51.886487 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:52.107431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:52.107398 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:52.424534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:52.424494 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:26:52.424723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:52.424551 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 13:26:53.425514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:53.425480 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:26:53.426045 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:53.425546 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 13:26:54.468344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:54.452537 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:26:54.468344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:54.452612 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 13:26:54.513627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:54.513541 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:54.993524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:54.993479 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:26:55.425370 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:55.425337 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:26:55.425543 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:55.425392 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 13:26:56.316223 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:56.316188 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:56.424745 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:56.424704 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:26:56.424919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:56.424774 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 13:26:56.482743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:56.482691 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:56.653930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:56.653843 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:26:57.430083 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:57.430044 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:26:57.431150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:57.431128 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:26:58.724564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:26:58.724517 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:01.714348 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:01.714312 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:01.884830 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:01.884784 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:03.597452 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:03.597420 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:04.410216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:04.410179 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:08.091447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.091407 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv"] Nov 03 13:27:08.092329 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.092303 4924 cpu_manager.go:414] "containerMap: removing container" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" Nov 03 13:27:08.092454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.092330 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" Nov 03 13:27:08.092522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.092468 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="590429ba-2e0e-44c7-9635-4b60db71b247" containerName="istio-proxy" Nov 03 13:27:08.097095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.097063 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.148756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.148724 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5295ab29\"/\"openshift-service-ca.crt\"" Nov 03 13:27:08.152313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.152283 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv"] Nov 03 13:27:08.173091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.173040 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqtgb\" (UniqueName: \"kubernetes.io/projected/9542ec83-268f-48d4-a677-4b42c3ac9c5f-kube-api-access-xqtgb\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.173340 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.173315 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tmp-dir\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.173518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.173490 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-model-cache\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.173634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.173551 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-home\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.173634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.173619 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tls-certs\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.173755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.173670 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-dshm\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.174326 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.174295 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:08.192170 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.192141 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5295ab29\"/\"kube-root-ca.crt\"" Nov 03 13:27:08.195289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.195245 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5295ab29\"/\"default-dockercfg-ftgj2\"" Nov 03 13:27:08.195396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.195255 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5295ab29\"/\"e2e-lifecycle-model-5295ab29-kserve-self-signed-certs\"" Nov 03 13:27:08.281334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.281293 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-dshm\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.283026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.282949 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqtgb\" (UniqueName: \"kubernetes.io/projected/9542ec83-268f-48d4-a677-4b42c3ac9c5f-kube-api-access-xqtgb\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.283281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.283258 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tmp-dir\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.283493 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.283466 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-model-cache\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.283767 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.283739 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-home\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.283939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.283923 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tls-certs\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.285770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.284797 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tmp-dir\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.285770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.285414 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-model-cache\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.285770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.285654 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-home\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.310195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.310021 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tls-certs\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.310565 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.310530 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-dshm\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.348209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.348131 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqtgb\" (UniqueName: \"kubernetes.io/projected/9542ec83-268f-48d4-a677-4b42c3ac9c5f-kube-api-access-xqtgb\") pod \"e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.410588 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.410550 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:08.817555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:08.817527 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv"] Nov 03 13:27:08.818216 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:27:08.818190 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9542ec83_268f_48d4_a677_4b42c3ac9c5f.slice/crio-b57dff49c014a8e25aa4d7d7abe4d36da3d805ec5515f149555769ea6232e06c WatchSource:0}: Error finding container b57dff49c014a8e25aa4d7d7abe4d36da3d805ec5515f149555769ea6232e06c: Status 404 returned error can't find the container with id b57dff49c014a8e25aa4d7d7abe4d36da3d805ec5515f149555769ea6232e06c Nov 03 13:27:09.304128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:09.304054 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" event={"ID":"9542ec83-268f-48d4-a677-4b42c3ac9c5f","Type":"ContainerStarted","Data":"208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809"} Nov 03 13:27:09.304485 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:09.304147 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" event={"ID":"9542ec83-268f-48d4-a677-4b42c3ac9c5f","Type":"ContainerStarted","Data":"b57dff49c014a8e25aa4d7d7abe4d36da3d805ec5515f149555769ea6232e06c"} Nov 03 13:27:09.411205 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:09.411162 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:09.413663 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:09.413616 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" podStartSLOduration=1.413601705 podStartE2EDuration="1.413601705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:09.411872792 +0000 UTC m=+1362.606276386" watchObservedRunningTime="2026-11-03 13:27:09.413601705 +0000 UTC m=+1362.608005298" Nov 03 13:27:09.687048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:09.687011 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:11.942896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:11.942854 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2"] Nov 03 13:27:11.947244 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:11.947222 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036179 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036244 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036457 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036512 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a993b-ad5a-4066-b7b0-417607091a40-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036553 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036723 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036829 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a993b-ad5a-4066-b7b0-417607091a40-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036860 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.036919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.036881 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5dj\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-kube-api-access-dh5dj\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.047974 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.047946 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2"] Nov 03 13:27:12.138232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138180 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138250 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a993b-ad5a-4066-b7b0-417607091a40-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138292 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138349 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138366 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a993b-ad5a-4066-b7b0-417607091a40-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138386 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138415 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5dj\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-kube-api-access-dh5dj\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138493 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138536 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138801 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138715 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138944 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138913 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.138992 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.138958 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.139190 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.139165 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.139479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.139457 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a993b-ad5a-4066-b7b0-417607091a40-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.140778 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.140751 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a993b-ad5a-4066-b7b0-417607091a40-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.140897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.140878 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.192251 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.192216 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.192455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.192430 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5dj\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-kube-api-access-dh5dj\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.262055 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.261962 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:12.688407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.688377 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2"] Nov 03 13:27:12.688938 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:27:12.688912 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337a993b_ad5a_4066_b7b0_417607091a40.slice/crio-ada7af3714615e77433ef7441ae579e1984975bfd7b77858bb420fa11d3f4c37 WatchSource:0}: Error finding container ada7af3714615e77433ef7441ae579e1984975bfd7b77858bb420fa11d3f4c37: Status 404 returned error can't find the container with id ada7af3714615e77433ef7441ae579e1984975bfd7b77858bb420fa11d3f4c37 Nov 03 13:27:12.691484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.691453 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:27:12.691553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.691518 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:27:12.691553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:12.691546 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:27:13.363771 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:13.363727 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" event={"ID":"337a993b-ad5a-4066-b7b0-417607091a40","Type":"ContainerStarted","Data":"a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735"} Nov 03 13:27:13.364296 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:13.363796 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" event={"ID":"337a993b-ad5a-4066-b7b0-417607091a40","Type":"ContainerStarted","Data":"ada7af3714615e77433ef7441ae579e1984975bfd7b77858bb420fa11d3f4c37"} Nov 03 13:27:13.524158 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:13.524103 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podStartSLOduration=2.524059387 podStartE2EDuration="2.524059387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:13.523162342 +0000 UTC m=+1366.717565935" watchObservedRunningTime="2026-11-03 13:27:13.524059387 +0000 UTC m=+1366.718462975" Nov 03 13:27:14.262640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:14.262604 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:14.264212 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:14.264182 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:27:14.264343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:14.264236 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 13:27:14.898979 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:14.898936 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:15.262685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:15.262644 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:27:15.262852 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:15.262707 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 13:27:16.263089 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:16.263022 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:27:16.263597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:16.263133 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 13:27:17.262650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:17.262610 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:27:17.262847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:17.262682 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 13:27:18.262831 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:18.262785 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:27:18.263288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:18.262857 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 13:27:18.410754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:18.410716 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:18.423479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:18.423452 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:18.585264 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:27:18.585167 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:27:19.262981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:19.262929 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:27:19.263419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:19.263017 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 13:27:19.417494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:19.417390 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:27:20.200693 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:20.200660 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:20.279757 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:20.279720 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:20.280754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:20.280697 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:20.281651 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:20.281630 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:27:24.371489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:24.371454 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:25.686938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:25.686894 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:30.901879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:30.901841 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:35.734988 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.734952 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r"] Nov 03 13:27:35.739408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.739390 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.840321 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.840272 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-30074e-serving-cert\"" Nov 03 13:27:35.843831 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.843799 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r"] Nov 03 13:27:35.876499 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.876447 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.876719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.876609 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-openshift-service-ca\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.876719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.876706 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-metrics-certs\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.876857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.876776 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9ng\" (UniqueName: \"kubernetes.io/projected/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-kube-api-access-4v9ng\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.923274 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.923239 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:35.977850 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.977813 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-openshift-service-ca\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.978033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.977888 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-metrics-certs\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.978033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.977941 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9ng\" (UniqueName: \"kubernetes.io/projected/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-kube-api-access-4v9ng\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.978033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.978007 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.978254 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:27:35.978137 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-serving-cert: secret "maas-api-e2e-api-a-30074e-serving-cert" not found Nov 03 13:27:35.978254 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:27:35.978195 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls podName:621679fd-6f97-4deb-bc4e-7fc4e7d2ff89 nodeName:}" failed. No retries permitted until 2026-11-03 13:27:36.478180298 +0000 UTC m=+1389.672583870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls") pod "maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" (UID: "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89") : secret "maas-api-e2e-api-a-30074e-serving-cert" not found Nov 03 13:27:35.978532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.978512 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-openshift-service-ca\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:35.980414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:35.980395 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-metrics-certs\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:36.037013 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:36.036922 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9ng\" (UniqueName: \"kubernetes.io/projected/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-kube-api-access-4v9ng\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:36.088662 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:36.088610 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:36.220798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:36.220767 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:36.487855 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:36.487820 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:36.490261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:36.490231 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls\") pod \"maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:36.654356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:36.654320 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:37.025529 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:37.025498 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r"] Nov 03 13:27:37.025988 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:27:37.025962 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621679fd_6f97_4deb_bc4e_7fc4e7d2ff89.slice/crio-0875d99e99ffd5ce14c25b97899316e65bedcfe2d2cbba321d0df3cbc5312dfa WatchSource:0}: Error finding container 0875d99e99ffd5ce14c25b97899316e65bedcfe2d2cbba321d0df3cbc5312dfa: Status 404 returned error can't find the container with id 0875d99e99ffd5ce14c25b97899316e65bedcfe2d2cbba321d0df3cbc5312dfa Nov 03 13:27:37.541280 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:37.541245 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" event={"ID":"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89","Type":"ContainerStarted","Data":"0875d99e99ffd5ce14c25b97899316e65bedcfe2d2cbba321d0df3cbc5312dfa"} Nov 03 13:27:38.549621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:38.549580 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" event={"ID":"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89","Type":"ContainerStarted","Data":"b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9"} Nov 03 13:27:38.550044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:38.549816 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:38.663327 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:38.663271 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" podStartSLOduration=2.690196486 podStartE2EDuration="3.66325673s" totalImagesPullingTime="973.060244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:38.660949278 +0000 UTC m=+1391.855352866" watchObservedRunningTime="2026-11-03 13:27:38.66325673 +0000 UTC m=+1391.857660324" Nov 03 13:27:41.404482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:41.404451 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:44.562746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:44.562714 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:27:46.593533 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:46.593495 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:48.560539 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:27:48.560497 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:27:51.739842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:51.739806 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:52.664880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:52.664834 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:53.839671 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:53.839621 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:54.105587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:54.105490 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:57.546065 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:57.546030 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:57.987623 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:57.987588 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:58.167552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:58.167514 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:27:58.330762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:27:58.330676 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:01.312735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:01.312698 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:01.492637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:01.490192 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:03.542746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:03.542705 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:08.505506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:08.505473 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:08.916796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:08.916763 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:14.122376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:14.122337 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:15.671504 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:15.671467 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:18.561175 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:28:18.561131 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:28:19.367411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:19.367371 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:24.700383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:24.700349 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:24.928341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:24.928287 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:28.017037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:28.016949 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:28.289834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:28.289745 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:28.441675 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:28.441626 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:28.821172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:28.821124 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-shvr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:28:28.821477 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:28.821194 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:28:28.956445 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:28.956413 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:29.713263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:29.713223 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:29.997513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:29.997412 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:30.929859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:30.929795 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:34.720277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:34.720243 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:34.922367 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:34.922328 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:35.116526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:35.116441 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:36.886336 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:36.886305 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:37.036588 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:37.036558 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:38.129523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:38.129488 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:38.316011 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:38.315958 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:41.773894 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:41.773863 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:41.913475 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:41.913441 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:42.054710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.054630 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:42.261984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.261946 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb"] Nov 03 13:28:42.267197 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.267170 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.333419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.333329 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdg74\" (UniqueName: \"kubernetes.io/projected/4b8aaf08-9c83-4051-ae05-48c4eae0b468-kube-api-access-xdg74\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.333419 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.333411 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.333647 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.333498 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-metrics-certs\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.333713 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.333696 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8aaf08-9c83-4051-ae05-48c4eae0b468-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.363234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.363205 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1fdd09-serving-cert\"" Nov 03 13:28:42.366963 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.366935 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb"] Nov 03 13:28:42.434522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.434492 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdg74\" (UniqueName: \"kubernetes.io/projected/4b8aaf08-9c83-4051-ae05-48c4eae0b468-kube-api-access-xdg74\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.434687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.434537 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.434687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.434570 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-metrics-certs\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.434687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.434674 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8aaf08-9c83-4051-ae05-48c4eae0b468-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.434826 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:28:42.434695 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-serving-cert: secret "maas-api-e2e-api-a-1fdd09-serving-cert" not found Nov 03 13:28:42.434826 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:28:42.434761 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls podName:4b8aaf08-9c83-4051-ae05-48c4eae0b468 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:42.934745774 +0000 UTC m=+1456.129149347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls") pod "maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" (UID: "4b8aaf08-9c83-4051-ae05-48c4eae0b468") : secret "maas-api-e2e-api-a-1fdd09-serving-cert" not found Nov 03 13:28:42.435328 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.435303 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8aaf08-9c83-4051-ae05-48c4eae0b468-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.437011 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.436984 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-metrics-certs\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.497308 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.497275 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdg74\" (UniqueName: \"kubernetes.io/projected/4b8aaf08-9c83-4051-ae05-48c4eae0b468-kube-api-access-xdg74\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.518169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.518136 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:42.820167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.820133 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:42.957612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.957572 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:42.960249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:42.960218 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls\") pod \"maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:43.181195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:43.181152 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:43.554860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:43.554828 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb"] Nov 03 13:28:43.555924 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:28:43.555893 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8aaf08_9c83_4051_ae05_48c4eae0b468.slice/crio-194d640d6faad9831e5f4f60dea0240fd32395b53ae8d8dea50333ba23f2aa65 WatchSource:0}: Error finding container 194d640d6faad9831e5f4f60dea0240fd32395b53ae8d8dea50333ba23f2aa65: Status 404 returned error can't find the container with id 194d640d6faad9831e5f4f60dea0240fd32395b53ae8d8dea50333ba23f2aa65 Nov 03 13:28:44.103356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:44.103320 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" event={"ID":"4b8aaf08-9c83-4051-ae05-48c4eae0b468","Type":"ContainerStarted","Data":"194d640d6faad9831e5f4f60dea0240fd32395b53ae8d8dea50333ba23f2aa65"} Nov 03 13:28:45.111854 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:45.111812 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" event={"ID":"4b8aaf08-9c83-4051-ae05-48c4eae0b468","Type":"ContainerStarted","Data":"85891d052f3b085014e9350d713439414397225504f69937141a252e24d699a8"} Nov 03 13:28:45.112342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:45.111976 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:45.225534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:45.225478 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" podStartSLOduration=2.305459409 podStartE2EDuration="3.22546443s" totalImagesPullingTime="920.005021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:45.222656365 +0000 UTC m=+1458.417059984" watchObservedRunningTime="2026-11-03 13:28:45.22546443 +0000 UTC m=+1458.419868024" Nov 03 13:28:47.968294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:47.968258 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:48.562472 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:28:48.559414 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:28:51.123959 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:51.123931 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:28:53.242538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:53.231511 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:58.534671 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:58.534634 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:28:59.016690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.016654 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-149af7"] Nov 03 13:28:59.020813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.020788 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-149af7" Nov 03 13:28:59.053219 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.053181 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pdd\" (UniqueName: \"kubernetes.io/projected/2c90ec1f-f378-4549-8e95-7e6eb3ec55d5-kube-api-access-z7pdd\") pod \"test-curl-7603-149af7\" (UID: \"2c90ec1f-f378-4549-8e95-7e6eb3ec55d5\") " pod="opendatahub/test-curl-7603-149af7" Nov 03 13:28:59.077798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.077761 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-149af7"] Nov 03 13:28:59.118646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.118613 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kmbtk\"" Nov 03 13:28:59.155046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.155013 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pdd\" (UniqueName: \"kubernetes.io/projected/2c90ec1f-f378-4549-8e95-7e6eb3ec55d5-kube-api-access-z7pdd\") pod \"test-curl-7603-149af7\" (UID: \"2c90ec1f-f378-4549-8e95-7e6eb3ec55d5\") " pod="opendatahub/test-curl-7603-149af7" Nov 03 13:28:59.209981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.209926 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pdd\" (UniqueName: \"kubernetes.io/projected/2c90ec1f-f378-4549-8e95-7e6eb3ec55d5-kube-api-access-z7pdd\") pod \"test-curl-7603-149af7\" (UID: \"2c90ec1f-f378-4549-8e95-7e6eb3ec55d5\") " pod="opendatahub/test-curl-7603-149af7" Nov 03 13:28:59.333652 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.333548 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-149af7" Nov 03 13:28:59.703869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:28:59.703840 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-149af7"] Nov 03 13:28:59.705123 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:28:59.705088 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c90ec1f_f378_4549_8e95_7e6eb3ec55d5.slice/crio-ce6bb58f8843710cdec049b2b0cae2421d4898aeeb501cd4625ef00e089d5863 WatchSource:0}: Error finding container ce6bb58f8843710cdec049b2b0cae2421d4898aeeb501cd4625ef00e089d5863: Status 404 returned error can't find the container with id ce6bb58f8843710cdec049b2b0cae2421d4898aeeb501cd4625ef00e089d5863 Nov 03 13:29:00.288047 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:00.288004 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-149af7" event={"ID":"2c90ec1f-f378-4549-8e95-7e6eb3ec55d5","Type":"ContainerStarted","Data":"ce6bb58f8843710cdec049b2b0cae2421d4898aeeb501cd4625ef00e089d5863"} Nov 03 13:29:02.302745 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:02.302706 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-149af7" event={"ID":"2c90ec1f-f378-4549-8e95-7e6eb3ec55d5","Type":"ContainerStarted","Data":"2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef"} Nov 03 13:29:02.411915 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:02.411862 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7603-149af7" podStartSLOduration=2.812815507 podStartE2EDuration="4.411847309s" totalImagesPullingTime="1.599031802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:02.408970352 +0000 UTC m=+1475.603373946" watchObservedRunningTime="2026-11-03 13:29:02.411847309 +0000 UTC m=+1475.606250902" Nov 03 13:29:03.584446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:03.584401 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:03.917599 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:03.917566 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-149af7"] Nov 03 13:29:03.921527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:03.921500 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-149af7"] Nov 03 13:29:04.114040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.114001 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2"] Nov 03 13:29:04.114747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.114729 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2c90ec1f-f378-4549-8e95-7e6eb3ec55d5" containerName="test-curl-7603-149af7" Nov 03 13:29:04.114811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.114749 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c90ec1f-f378-4549-8e95-7e6eb3ec55d5" containerName="test-curl-7603-149af7" Nov 03 13:29:04.114865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.114856 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c90ec1f-f378-4549-8e95-7e6eb3ec55d5" containerName="test-curl-7603-149af7" Nov 03 13:29:04.118858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.118841 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.169350 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.169253 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"openshift-service-ca.crt\"" Nov 03 13:29:04.172511 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.172486 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2"] Nov 03 13:29:04.214829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.214794 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"kube-root-ca.crt\"" Nov 03 13:29:04.214829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.214815 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"distinct-w2-cdb532-kserve-self-signed-certs\"" Nov 03 13:29:04.215026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.214802 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"default-dockercfg-zprwm\"" Nov 03 13:29:04.217649 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.217630 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-dshm\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.217721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.217671 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkvt\" (UniqueName: \"kubernetes.io/projected/723ac416-02d4-48a5-a50b-21fb78a3fce5-kube-api-access-zkkvt\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.217721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.217700 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-tmp-dir\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.217848 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.217828 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-model-cache\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.218001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.217982 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723ac416-02d4-48a5-a50b-21fb78a3fce5-tls-certs\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.218049 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.218034 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-home\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.321348 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.321310 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7603-149af7" podUID="2c90ec1f-f378-4549-8e95-7e6eb3ec55d5" containerName="test-curl-7603-149af7" containerID="cri-o://2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef" gracePeriod=2 Nov 03 13:29:04.323529 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.323492 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkvt\" (UniqueName: \"kubernetes.io/projected/723ac416-02d4-48a5-a50b-21fb78a3fce5-kube-api-access-zkkvt\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.324102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.324064 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-tmp-dir\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.326215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.326191 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-model-cache\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.326917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.326883 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723ac416-02d4-48a5-a50b-21fb78a3fce5-tls-certs\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.327235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.326964 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-home\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.327838 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.326658 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-model-cache\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.327938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.324598 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-tmp-dir\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.329465 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.329429 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-home\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.329595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.329576 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-dshm\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.332060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.332034 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723ac416-02d4-48a5-a50b-21fb78a3fce5-tls-certs\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.332209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.332114 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-dshm\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.369442 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:29:04.369403 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7603-149af7\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="2c90ec1f-f378-4549-8e95-7e6eb3ec55d5" pod="opendatahub/test-curl-7603-149af7" Nov 03 13:29:04.377751 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.377719 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkvt\" (UniqueName: \"kubernetes.io/projected/723ac416-02d4-48a5-a50b-21fb78a3fce5-kube-api-access-zkkvt\") pod \"distinct-w2-cdb532-kserve-6565dcd496-pbcn2\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.431492 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.431400 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:04.872475 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.872375 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:04.911342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:04.911312 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2"] Nov 03 13:29:04.912220 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:29:04.912195 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod723ac416_02d4_48a5_a50b_21fb78a3fce5.slice/crio-a4919900c14c3ea7cd3666ad792ba7290f4c52795cc8ad62daf7cc615da7bb41 WatchSource:0}: Error finding container a4919900c14c3ea7cd3666ad792ba7290f4c52795cc8ad62daf7cc615da7bb41: Status 404 returned error can't find the container with id a4919900c14c3ea7cd3666ad792ba7290f4c52795cc8ad62daf7cc615da7bb41 Nov 03 13:29:05.327763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:05.327725 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" event={"ID":"723ac416-02d4-48a5-a50b-21fb78a3fce5","Type":"ContainerStarted","Data":"3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a"} Nov 03 13:29:05.327763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:05.327769 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" event={"ID":"723ac416-02d4-48a5-a50b-21fb78a3fce5","Type":"ContainerStarted","Data":"a4919900c14c3ea7cd3666ad792ba7290f4c52795cc8ad62daf7cc615da7bb41"} Nov 03 13:29:05.431530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:05.431497 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:05.437479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:05.437434 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" podStartSLOduration=1.437422092 podStartE2EDuration="1.437422092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:05.43553449 +0000 UTC m=+1478.629938084" watchObservedRunningTime="2026-11-03 13:29:05.437422092 +0000 UTC m=+1478.631825685" Nov 03 13:29:06.573521 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:06.573495 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-149af7" Nov 03 13:29:06.658432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:06.658406 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c90ec1f-f378-4549-8e95-7e6eb3ec55d5-kube-api-access-z7pdd\" (UniqueName: \"kubernetes.io/projected/2c90ec1f-f378-4549-8e95-7e6eb3ec55d5-kube-api-access-z7pdd\") pod \"2c90ec1f-f378-4549-8e95-7e6eb3ec55d5\" (UID: \"2c90ec1f-f378-4549-8e95-7e6eb3ec55d5\") " Nov 03 13:29:06.660577 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:06.660548 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c90ec1f-f378-4549-8e95-7e6eb3ec55d5-kube-api-access-z7pdd" pod "2c90ec1f-f378-4549-8e95-7e6eb3ec55d5" (UID: "2c90ec1f-f378-4549-8e95-7e6eb3ec55d5"). InnerVolumeSpecName "kube-api-access-z7pdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:06.759549 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:06.759510 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pdd\" (UniqueName: \"kubernetes.io/projected/2c90ec1f-f378-4549-8e95-7e6eb3ec55d5-kube-api-access-z7pdd\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:06.951992 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:06.951865 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c90ec1f-f378-4549-8e95-7e6eb3ec55d5" path="/var/lib/kubelet/pods/2c90ec1f-f378-4549-8e95-7e6eb3ec55d5/volumes" Nov 03 13:29:07.344844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:07.344747 4924 generic.go:356] "Generic (PLEG): container finished" podID="2c90ec1f-f378-4549-8e95-7e6eb3ec55d5" containerID="2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef" exitCode=137 Nov 03 13:29:07.344844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:07.344795 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-149af7" Nov 03 13:29:07.345098 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:07.344856 4924 scope.go:122] "RemoveContainer" containerID="2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef" Nov 03 13:29:07.354746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:07.354724 4924 scope.go:122] "RemoveContainer" containerID="2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef" Nov 03 13:29:07.355066 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:29:07.355037 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef\": container with ID starting with 2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef not found: ID does not exist" containerID="2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef" Nov 03 13:29:07.355191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:07.355086 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef"} err="failed to get container status \"2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef\": rpc error: code = NotFound desc = could not find container \"2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef\": container with ID starting with 2ec9a01229537deb196317b16141f2e428c394b46fdcc6649f3c7a244e75c4ef not found: ID does not exist" Nov 03 13:29:09.208199 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:09.208153 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:10.533572 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:10.533531 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:14.444093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:14.432232 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:14.451913 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:14.451885 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:14.502054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:14.502000 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:15.425168 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:15.425137 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 13:29:16.988197 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:16.988158 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:17.144521 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:17.144478 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:18.576288 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:29:18.576248 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:29:19.905934 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:19.905900 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:24.723140 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.723102 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67"] Nov 03 13:29:24.728007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.727985 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.779974 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.779915 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-abe0e0-serving-cert\"" Nov 03 13:29:24.806351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.806306 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-metrics-certs\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.806551 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.806447 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369abf4-255b-4b8f-8393-61973d08a0df-openshift-service-ca\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.806618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.806569 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppsm\" (UniqueName: \"kubernetes.io/projected/7369abf4-255b-4b8f-8393-61973d08a0df-kube-api-access-vppsm\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.806618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.806599 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.830303 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.830264 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67"] Nov 03 13:29:24.901872 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.901828 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:24.908218 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.908185 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369abf4-255b-4b8f-8393-61973d08a0df-openshift-service-ca\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.908411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.908295 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppsm\" (UniqueName: \"kubernetes.io/projected/7369abf4-255b-4b8f-8393-61973d08a0df-kube-api-access-vppsm\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.908411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.908333 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.908578 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.908441 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-metrics-certs\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.908578 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:29:24.908559 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-serving-cert: secret "maas-api-e2e-api-b-abe0e0-serving-cert" not found Nov 03 13:29:24.908722 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:29:24.908648 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls podName:7369abf4-255b-4b8f-8393-61973d08a0df nodeName:}" failed. No retries permitted until 2026-11-03 13:29:25.40862461 +0000 UTC m=+1498.603028187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls") pod "maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" (UID: "7369abf4-255b-4b8f-8393-61973d08a0df") : secret "maas-api-e2e-api-b-abe0e0-serving-cert" not found Nov 03 13:29:24.909061 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.909036 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369abf4-255b-4b8f-8393-61973d08a0df-openshift-service-ca\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.911231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.911205 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-metrics-certs\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:24.963397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:24.963368 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppsm\" (UniqueName: \"kubernetes.io/projected/7369abf4-255b-4b8f-8393-61973d08a0df-kube-api-access-vppsm\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:25.037369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:25.037273 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:25.160586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:25.160551 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:25.288207 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:25.288119 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:25.413826 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:25.413790 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:25.416494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:25.416459 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls\") pod \"maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:25.642286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:25.642192 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:26.017123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:26.017092 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67"] Nov 03 13:29:26.018016 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:29:26.017989 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7369abf4_255b_4b8f_8393_61973d08a0df.slice/crio-91803b669643138b650dd4abc985e1a61f95970de8f2aff90d3b4703254ab7d5 WatchSource:0}: Error finding container 91803b669643138b650dd4abc985e1a61f95970de8f2aff90d3b4703254ab7d5: Status 404 returned error can't find the container with id 91803b669643138b650dd4abc985e1a61f95970de8f2aff90d3b4703254ab7d5 Nov 03 13:29:26.520148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:26.520100 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" event={"ID":"7369abf4-255b-4b8f-8393-61973d08a0df","Type":"ContainerStarted","Data":"91803b669643138b650dd4abc985e1a61f95970de8f2aff90d3b4703254ab7d5"} Nov 03 13:29:27.172351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:27.172314 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:29:27.173648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:27.173626 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:29:27.529171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:27.529090 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" event={"ID":"7369abf4-255b-4b8f-8393-61973d08a0df","Type":"ContainerStarted","Data":"8327511c85da0e96f1ca8aa8e1a049edccaf2af070466c4bf188d905a6b14ccb"} Nov 03 13:29:27.529319 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:27.529242 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:27.642784 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:27.642729 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" podStartSLOduration=2.728975836 podStartE2EDuration="3.642714304s" totalImagesPullingTime="913.738468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:27.64044246 +0000 UTC m=+1500.834846054" watchObservedRunningTime="2026-11-03 13:29:27.642714304 +0000 UTC m=+1500.837117898" Nov 03 13:29:30.391443 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:30.391409 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:33.542768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:33.542737 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:29:35.515098 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:35.515039 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:39.380575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:39.380538 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:40.481015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:40.480972 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:40.756537 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:40.756438 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:40.943387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:40.943350 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:44.783063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:44.783034 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:44.976724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:44.976691 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:45.128576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:45.128478 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:45.307468 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:45.307433 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:45.953586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:45.953543 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:46.596422 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:46.596319 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:46.748910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:46.748836 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:47.252707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:47.252671 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:48.090434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:48.090397 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:48.271606 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:48.271557 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:48.561216 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:29:48.561179 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:29:51.277689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:51.277648 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:51.449518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:51.449465 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:51.592838 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:51.592757 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:53.251803 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.251761 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:53.427787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.427756 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc"] Nov 03 13:29:53.431931 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.431906 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.506968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.506881 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-home\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.506968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.506949 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-tmp-dir\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.507209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.507067 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a983cf0a-e87a-4615-a048-df14fb6c2f45-tls-certs\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.507209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.507180 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-dshm\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.507334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.507316 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-model-cache\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.507410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.507396 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjl4\" (UniqueName: \"kubernetes.io/projected/a983cf0a-e87a-4615-a048-df14fb6c2f45-kube-api-access-9bjl4\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.510395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.510370 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:53.527580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.527545 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"distinct-2-w2-cdb532-kserve-self-signed-certs\"" Nov 03 13:29:53.533434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.533400 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc"] Nov 03 13:29:53.609890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.609834 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-home\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.610115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.609921 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-tmp-dir\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.610115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.609994 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a983cf0a-e87a-4615-a048-df14fb6c2f45-tls-certs\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.610115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.610040 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-dshm\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.610264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.610180 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-model-cache\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.610264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.610251 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjl4\" (UniqueName: \"kubernetes.io/projected/a983cf0a-e87a-4615-a048-df14fb6c2f45-kube-api-access-9bjl4\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.610982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.610954 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-home\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.611251 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.611226 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-tmp-dir\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.614528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.614495 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-model-cache\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.614900 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.614874 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a983cf0a-e87a-4615-a048-df14fb6c2f45-tls-certs\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.615853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.615824 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-dshm\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.664725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.664691 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjl4\" (UniqueName: \"kubernetes.io/projected/a983cf0a-e87a-4615-a048-df14fb6c2f45-kube-api-access-9bjl4\") pod \"distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:53.745406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:53.745371 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:54.132739 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:54.132711 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc"] Nov 03 13:29:54.134051 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:29:54.134027 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda983cf0a_e87a_4615_a048_df14fb6c2f45.slice/crio-a1c5321c802c4cac4f8f63ffab7d668aa8bc42c4ce39ce31d115e9742946ef54 WatchSource:0}: Error finding container a1c5321c802c4cac4f8f63ffab7d668aa8bc42c4ce39ce31d115e9742946ef54: Status 404 returned error can't find the container with id a1c5321c802c4cac4f8f63ffab7d668aa8bc42c4ce39ce31d115e9742946ef54 Nov 03 13:29:54.590560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:54.590523 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:54.774942 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:54.774909 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:29:54.791955 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:54.791917 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" event={"ID":"a983cf0a-e87a-4615-a048-df14fb6c2f45","Type":"ContainerStarted","Data":"e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b"} Nov 03 13:29:54.792136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:54.791966 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" event={"ID":"a983cf0a-e87a-4615-a048-df14fb6c2f45","Type":"ContainerStarted","Data":"a1c5321c802c4cac4f8f63ffab7d668aa8bc42c4ce39ce31d115e9742946ef54"} Nov 03 13:29:54.904429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:54.904323 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" podStartSLOduration=1.90429969 podStartE2EDuration="1.90429969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:54.902340284 +0000 UTC m=+1528.096743880" watchObservedRunningTime="2026-11-03 13:29:54.90429969 +0000 UTC m=+1528.098703287" Nov 03 13:29:55.746276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:55.746236 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:29:59.261298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:29:59.261263 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:00.170899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.170858 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr"] Nov 03 13:30:00.175589 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.175566 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:00.212849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.212818 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm"] Nov 03 13:30:00.217276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.217251 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:00.227272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.227245 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:30:00.261529 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.261490 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcfb\" (UniqueName: \"kubernetes.io/projected/97dcb221-860c-4bf0-8a18-9077c35dcfe9-kube-api-access-mdcfb\") pod \"maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm\" (UID: \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:00.261925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.261581 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97dcb221-860c-4bf0-8a18-9077c35dcfe9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm\" (UID: \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:00.261925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.261756 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnxw\" (UniqueName: \"kubernetes.io/projected/6c6cb512-4454-4687-8487-3d03d6006b75-kube-api-access-5qnxw\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr\" (UID: \"6c6cb512-4454-4687-8487-3d03d6006b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:00.261925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.261856 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb512-4454-4687-8487-3d03d6006b75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr\" (UID: \"6c6cb512-4454-4687-8487-3d03d6006b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:00.271217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.271183 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr"] Nov 03 13:30:00.335517 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.335481 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm"] Nov 03 13:30:00.363930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.363887 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcfb\" (UniqueName: \"kubernetes.io/projected/97dcb221-860c-4bf0-8a18-9077c35dcfe9-kube-api-access-mdcfb\") pod \"maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm\" (UID: \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:00.364244 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.364224 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97dcb221-860c-4bf0-8a18-9077c35dcfe9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm\" (UID: \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:00.364510 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.364495 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnxw\" (UniqueName: \"kubernetes.io/projected/6c6cb512-4454-4687-8487-3d03d6006b75-kube-api-access-5qnxw\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr\" (UID: \"6c6cb512-4454-4687-8487-3d03d6006b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:00.364679 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.364665 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb512-4454-4687-8487-3d03d6006b75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr\" (UID: \"6c6cb512-4454-4687-8487-3d03d6006b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:00.367342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.367313 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb512-4454-4687-8487-3d03d6006b75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr\" (UID: \"6c6cb512-4454-4687-8487-3d03d6006b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:00.368353 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.368334 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97dcb221-860c-4bf0-8a18-9077c35dcfe9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm\" (UID: \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:00.430557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.430481 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnxw\" (UniqueName: \"kubernetes.io/projected/6c6cb512-4454-4687-8487-3d03d6006b75-kube-api-access-5qnxw\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr\" (UID: \"6c6cb512-4454-4687-8487-3d03d6006b75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:00.432060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.432034 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcfb\" (UniqueName: \"kubernetes.io/projected/97dcb221-860c-4bf0-8a18-9077c35dcfe9-kube-api-access-mdcfb\") pod \"maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm\" (UID: \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:00.490388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.490348 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:00.531234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.531198 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:00.865849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.865766 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr"] Nov 03 13:30:00.903293 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:30:00.903255 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6cb512_4454_4687_8487_3d03d6006b75.slice/crio-317508f32758496f70506b9f4a3d8e30bb2e417e86b39229c14741a97e82e04c WatchSource:0}: Error finding container 317508f32758496f70506b9f4a3d8e30bb2e417e86b39229c14741a97e82e04c: Status 404 returned error can't find the container with id 317508f32758496f70506b9f4a3d8e30bb2e417e86b39229c14741a97e82e04c Nov 03 13:30:00.966387 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:30:00.966342 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dcb221_860c_4bf0_8a18_9077c35dcfe9.slice/crio-1fbfea002cf1377c8a180e875c78bdc9ee4e12888545fa6bd2c761223d6541da WatchSource:0}: Error finding container 1fbfea002cf1377c8a180e875c78bdc9ee4e12888545fa6bd2c761223d6541da: Status 404 returned error can't find the container with id 1fbfea002cf1377c8a180e875c78bdc9ee4e12888545fa6bd2c761223d6541da Nov 03 13:30:00.967301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:00.967277 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm"] Nov 03 13:30:01.843248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:01.843207 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" event={"ID":"97dcb221-860c-4bf0-8a18-9077c35dcfe9","Type":"ContainerStarted","Data":"29c0fc9761dff2af67eda1772938e1491c7b423ab12f9be31f9c9e750ca464b0"} Nov 03 13:30:01.843711 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:01.843279 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" event={"ID":"97dcb221-860c-4bf0-8a18-9077c35dcfe9","Type":"ContainerStarted","Data":"1fbfea002cf1377c8a180e875c78bdc9ee4e12888545fa6bd2c761223d6541da"} Nov 03 13:30:01.845146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:01.845118 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" event={"ID":"6c6cb512-4454-4687-8487-3d03d6006b75","Type":"ContainerStarted","Data":"5036bf024a45d9477f53ac521911b619624256f8bc12ae506ac71a19f07551cb"} Nov 03 13:30:01.845264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:01.845175 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" event={"ID":"6c6cb512-4454-4687-8487-3d03d6006b75","Type":"ContainerStarted","Data":"317508f32758496f70506b9f4a3d8e30bb2e417e86b39229c14741a97e82e04c"} Nov 03 13:30:01.953316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:01.953270 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" podStartSLOduration=1.9532571079999999 podStartE2EDuration="1.953257108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:01.951290016 +0000 UTC m=+1535.145693609" watchObservedRunningTime="2026-11-03 13:30:01.953257108 +0000 UTC m=+1535.147660702" Nov 03 13:30:02.060178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:02.060114 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" podStartSLOduration=2.060095707 podStartE2EDuration="2.060095707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:02.05996274 +0000 UTC m=+1535.254366338" watchObservedRunningTime="2026-11-03 13:30:02.060095707 +0000 UTC m=+1535.254499299" Nov 03 13:30:02.858053 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:02.858014 4924 generic.go:356] "Generic (PLEG): container finished" podID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerID="29c0fc9761dff2af67eda1772938e1491c7b423ab12f9be31f9c9e750ca464b0" exitCode=60 Nov 03 13:30:02.858561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:02.858108 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" event={"ID":"97dcb221-860c-4bf0-8a18-9077c35dcfe9","Type":"ContainerDied","Data":"29c0fc9761dff2af67eda1772938e1491c7b423ab12f9be31f9c9e750ca464b0"} Nov 03 13:30:02.858561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:02.858498 4924 scope.go:122] "RemoveContainer" containerID="29c0fc9761dff2af67eda1772938e1491c7b423ab12f9be31f9c9e750ca464b0" Nov 03 13:30:02.860017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:02.859996 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c6cb512-4454-4687-8487-3d03d6006b75" containerID="5036bf024a45d9477f53ac521911b619624256f8bc12ae506ac71a19f07551cb" exitCode=60 Nov 03 13:30:02.860171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:02.860155 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" event={"ID":"6c6cb512-4454-4687-8487-3d03d6006b75","Type":"ContainerDied","Data":"5036bf024a45d9477f53ac521911b619624256f8bc12ae506ac71a19f07551cb"} Nov 03 13:30:02.860461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:02.860446 4924 scope.go:122] "RemoveContainer" containerID="5036bf024a45d9477f53ac521911b619624256f8bc12ae506ac71a19f07551cb" Nov 03 13:30:03.746150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:03.746110 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:30:03.759719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:03.759684 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:30:03.871445 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:03.871397 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" event={"ID":"97dcb221-860c-4bf0-8a18-9077c35dcfe9","Type":"ContainerStarted","Data":"8e762775ad13bf943dbe830bf6dcd066b839eb310d0bc32874016eade006d522"} Nov 03 13:30:03.873754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:03.873720 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" event={"ID":"6c6cb512-4454-4687-8487-3d03d6006b75","Type":"ContainerStarted","Data":"3e5a50923e2a180f55a09db0ccfae61a46bec1e443fc4ebb567170366a4efe54"} Nov 03 13:30:03.885668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:03.885645 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 13:30:04.499744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.499709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:04.884208 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.884120 4924 generic.go:356] "Generic (PLEG): container finished" podID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerID="8e762775ad13bf943dbe830bf6dcd066b839eb310d0bc32874016eade006d522" exitCode=60 Nov 03 13:30:04.884644 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.884223 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" event={"ID":"97dcb221-860c-4bf0-8a18-9077c35dcfe9","Type":"ContainerDied","Data":"8e762775ad13bf943dbe830bf6dcd066b839eb310d0bc32874016eade006d522"} Nov 03 13:30:04.884644 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.884281 4924 scope.go:122] "RemoveContainer" containerID="29c0fc9761dff2af67eda1772938e1491c7b423ab12f9be31f9c9e750ca464b0" Nov 03 13:30:04.884644 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.884496 4924 scope.go:122] "RemoveContainer" containerID="8e762775ad13bf943dbe830bf6dcd066b839eb310d0bc32874016eade006d522" Nov 03 13:30:04.884863 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:04.884841 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm_odh-ai-gateway-infra(97dcb221-860c-4bf0-8a18-9077c35dcfe9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" Nov 03 13:30:04.886642 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.886619 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c6cb512-4454-4687-8487-3d03d6006b75" containerID="3e5a50923e2a180f55a09db0ccfae61a46bec1e443fc4ebb567170366a4efe54" exitCode=60 Nov 03 13:30:04.886748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.886703 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" event={"ID":"6c6cb512-4454-4687-8487-3d03d6006b75","Type":"ContainerDied","Data":"3e5a50923e2a180f55a09db0ccfae61a46bec1e443fc4ebb567170366a4efe54"} Nov 03 13:30:04.886935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.886921 4924 scope.go:122] "RemoveContainer" containerID="3e5a50923e2a180f55a09db0ccfae61a46bec1e443fc4ebb567170366a4efe54" Nov 03 13:30:04.887222 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:04.887203 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr_odh-ai-gateway-infra(6c6cb512-4454-4687-8487-3d03d6006b75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" Nov 03 13:30:04.897216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:04.897195 4924 scope.go:122] "RemoveContainer" containerID="5036bf024a45d9477f53ac521911b619624256f8bc12ae506ac71a19f07551cb" Nov 03 13:30:05.895236 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:05.895209 4924 scope.go:122] "RemoveContainer" containerID="8e762775ad13bf943dbe830bf6dcd066b839eb310d0bc32874016eade006d522" Nov 03 13:30:05.895703 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:05.895603 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm_odh-ai-gateway-infra(97dcb221-860c-4bf0-8a18-9077c35dcfe9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" Nov 03 13:30:05.896682 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:05.896660 4924 scope.go:122] "RemoveContainer" containerID="3e5a50923e2a180f55a09db0ccfae61a46bec1e443fc4ebb567170366a4efe54" Nov 03 13:30:05.896928 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:05.896910 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr_odh-ai-gateway-infra(6c6cb512-4454-4687-8487-3d03d6006b75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" Nov 03 13:30:09.773358 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:09.773324 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:14.894866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:14.894828 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:14.943732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:14.943699 4924 scope.go:122] "RemoveContainer" containerID="3e5a50923e2a180f55a09db0ccfae61a46bec1e443fc4ebb567170366a4efe54" Nov 03 13:30:15.944264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:15.944228 4924 scope.go:122] "RemoveContainer" containerID="8e762775ad13bf943dbe830bf6dcd066b839eb310d0bc32874016eade006d522" Nov 03 13:30:15.988624 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:15.988592 4924 generic.go:356] "Generic (PLEG): container finished" podID="6c6cb512-4454-4687-8487-3d03d6006b75" containerID="8ec72ffb03d6aa890872decd3f3fb4ac4ceb7fcbcde644102a1eb110ee276bf2" exitCode=60 Nov 03 13:30:15.988783 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:15.988668 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" event={"ID":"6c6cb512-4454-4687-8487-3d03d6006b75","Type":"ContainerDied","Data":"8ec72ffb03d6aa890872decd3f3fb4ac4ceb7fcbcde644102a1eb110ee276bf2"} Nov 03 13:30:15.988783 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:15.988728 4924 scope.go:122] "RemoveContainer" containerID="3e5a50923e2a180f55a09db0ccfae61a46bec1e443fc4ebb567170366a4efe54" Nov 03 13:30:15.989033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:15.989012 4924 scope.go:122] "RemoveContainer" containerID="8ec72ffb03d6aa890872decd3f3fb4ac4ceb7fcbcde644102a1eb110ee276bf2" Nov 03 13:30:15.989311 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:15.989287 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr_odh-ai-gateway-infra(6c6cb512-4454-4687-8487-3d03d6006b75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" Nov 03 13:30:16.997816 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:16.997777 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" event={"ID":"97dcb221-860c-4bf0-8a18-9077c35dcfe9","Type":"ContainerStarted","Data":"7460bfbf4b1b98ec6b2f712219d45fd2b0bcd1821142ffa21de4cc1f4cd83950"} Nov 03 13:30:17.115048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:17.115017 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr"] Nov 03 13:30:17.274311 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:17.274276 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:17.407062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:17.407028 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6cb512-4454-4687-8487-3d03d6006b75-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb512-4454-4687-8487-3d03d6006b75-maas-api-service-ca\") pod \"6c6cb512-4454-4687-8487-3d03d6006b75\" (UID: \"6c6cb512-4454-4687-8487-3d03d6006b75\") " Nov 03 13:30:17.407241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:17.407199 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6cb512-4454-4687-8487-3d03d6006b75-kube-api-access-5qnxw\" (UniqueName: \"kubernetes.io/projected/6c6cb512-4454-4687-8487-3d03d6006b75-kube-api-access-5qnxw\") pod \"6c6cb512-4454-4687-8487-3d03d6006b75\" (UID: \"6c6cb512-4454-4687-8487-3d03d6006b75\") " Nov 03 13:30:17.407474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:17.407447 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6cb512-4454-4687-8487-3d03d6006b75-maas-api-service-ca" pod "6c6cb512-4454-4687-8487-3d03d6006b75" (UID: "6c6cb512-4454-4687-8487-3d03d6006b75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:17.407722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:17.407701 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb512-4454-4687-8487-3d03d6006b75-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:17.424273 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:17.424240 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6cb512-4454-4687-8487-3d03d6006b75-kube-api-access-5qnxw" pod "6c6cb512-4454-4687-8487-3d03d6006b75" (UID: "6c6cb512-4454-4687-8487-3d03d6006b75"). InnerVolumeSpecName "kube-api-access-5qnxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:17.508946 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:17.508865 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnxw\" (UniqueName: \"kubernetes.io/projected/6c6cb512-4454-4687-8487-3d03d6006b75-kube-api-access-5qnxw\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:18.007731 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.007694 4924 generic.go:356] "Generic (PLEG): container finished" podID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerID="7460bfbf4b1b98ec6b2f712219d45fd2b0bcd1821142ffa21de4cc1f4cd83950" exitCode=60 Nov 03 13:30:18.008263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.007760 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" event={"ID":"97dcb221-860c-4bf0-8a18-9077c35dcfe9","Type":"ContainerDied","Data":"7460bfbf4b1b98ec6b2f712219d45fd2b0bcd1821142ffa21de4cc1f4cd83950"} Nov 03 13:30:18.008263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.007822 4924 scope.go:122] "RemoveContainer" containerID="8e762775ad13bf943dbe830bf6dcd066b839eb310d0bc32874016eade006d522" Nov 03 13:30:18.008263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.008253 4924 scope.go:122] "RemoveContainer" containerID="7460bfbf4b1b98ec6b2f712219d45fd2b0bcd1821142ffa21de4cc1f4cd83950" Nov 03 13:30:18.008619 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:18.008586 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm_odh-ai-gateway-infra(97dcb221-860c-4bf0-8a18-9077c35dcfe9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" Nov 03 13:30:18.009463 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.009440 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" Nov 03 13:30:18.009567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.009461 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr" event={"ID":"6c6cb512-4454-4687-8487-3d03d6006b75","Type":"ContainerDied","Data":"317508f32758496f70506b9f4a3d8e30bb2e417e86b39229c14741a97e82e04c"} Nov 03 13:30:18.022282 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.022263 4924 scope.go:122] "RemoveContainer" containerID="8ec72ffb03d6aa890872decd3f3fb4ac4ceb7fcbcde644102a1eb110ee276bf2" Nov 03 13:30:18.165365 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.165327 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm"] Nov 03 13:30:18.223601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.223564 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr"] Nov 03 13:30:18.235621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.235590 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895210-f9qlr"] Nov 03 13:30:18.559327 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:18.559290 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:30:18.743357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.743316 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:18.951870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:18.951837 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" path="/var/lib/kubelet/pods/6c6cb512-4454-4687-8487-3d03d6006b75/volumes" Nov 03 13:30:19.183474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:19.183446 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:19.331903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:19.331815 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dcb221-860c-4bf0-8a18-9077c35dcfe9-kube-api-access-mdcfb\" (UniqueName: \"kubernetes.io/projected/97dcb221-860c-4bf0-8a18-9077c35dcfe9-kube-api-access-mdcfb\") pod \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\" (UID: \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\") " Nov 03 13:30:19.332112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:19.332021 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97dcb221-860c-4bf0-8a18-9077c35dcfe9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97dcb221-860c-4bf0-8a18-9077c35dcfe9-maas-api-service-ca\") pod \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\" (UID: \"97dcb221-860c-4bf0-8a18-9077c35dcfe9\") " Nov 03 13:30:19.332506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:19.332461 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dcb221-860c-4bf0-8a18-9077c35dcfe9-maas-api-service-ca" pod "97dcb221-860c-4bf0-8a18-9077c35dcfe9" (UID: "97dcb221-860c-4bf0-8a18-9077c35dcfe9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:19.332647 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:19.332599 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97dcb221-860c-4bf0-8a18-9077c35dcfe9-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:19.334207 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:19.334182 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dcb221-860c-4bf0-8a18-9077c35dcfe9-kube-api-access-mdcfb" pod "97dcb221-860c-4bf0-8a18-9077c35dcfe9" (UID: "97dcb221-860c-4bf0-8a18-9077c35dcfe9"). InnerVolumeSpecName "kube-api-access-mdcfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:19.442184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:19.442115 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdcfb\" (UniqueName: \"kubernetes.io/projected/97dcb221-860c-4bf0-8a18-9077c35dcfe9-kube-api-access-mdcfb\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:20.028656 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:20.028618 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" event={"ID":"97dcb221-860c-4bf0-8a18-9077c35dcfe9","Type":"ContainerDied","Data":"1fbfea002cf1377c8a180e875c78bdc9ee4e12888545fa6bd2c761223d6541da"} Nov 03 13:30:20.028864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:20.028659 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm" Nov 03 13:30:20.028864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:20.028680 4924 scope.go:122] "RemoveContainer" containerID="7460bfbf4b1b98ec6b2f712219d45fd2b0bcd1821142ffa21de4cc1f4cd83950" Nov 03 13:30:20.185431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:20.185394 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm"] Nov 03 13:30:20.231598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:20.231561 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1fdd09-29895210-wwzkm"] Nov 03 13:30:20.369179 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:20.369103 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:20.951509 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:20.951476 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" path="/var/lib/kubelet/pods/97dcb221-860c-4bf0-8a18-9077c35dcfe9/volumes" Nov 03 13:30:22.009919 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:22.009880 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:23.807423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:23.807380 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:25.354382 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:25.354349 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:25.955972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:25.955934 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:28.163873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.163837 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8"] Nov 03 13:30:28.164720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164700 4924 cpu_manager.go:414] "containerMap: removing container" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.164774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164723 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.164774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164741 4924 cpu_manager.go:414] "containerMap: removing container" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.164774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164748 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.164774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164761 4924 cpu_manager.go:414] "containerMap: removing container" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.164774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164769 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.164916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164823 4924 cpu_manager.go:414] "containerMap: removing container" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.164916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164830 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.164998 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164973 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.164998 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164983 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.164998 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.164995 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.165142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.165002 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.165142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.165009 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.170370 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.170345 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.208599 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.208573 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw"] Nov 03 13:30:28.209199 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.209182 4924 cpu_manager.go:414] "containerMap: removing container" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.209261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.209201 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.209261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.209224 4924 cpu_manager.go:414] "containerMap: removing container" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.209261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.209229 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6cb512-4454-4687-8487-3d03d6006b75" containerName="cleanup" Nov 03 13:30:28.209412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.209332 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dcb221-860c-4bf0-8a18-9077c35dcfe9" containerName="cleanup" Nov 03 13:30:28.212528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.212512 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.222194 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.222172 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cdb532-openshift-default-dockercfg-q2fdz\"" Nov 03 13:30:28.253488 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.253447 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.253632 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.253614 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drgz\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-kube-api-access-2drgz\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.253745 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.253725 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fae73af-85a0-4129-909d-6be7992d360d-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.253821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.253806 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.253874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.253833 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.253983 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.253962 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.254044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.254019 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.254138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.254120 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.254173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.254151 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fae73af-85a0-4129-909d-6be7992d360d-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.264312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.264284 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8"] Nov 03 13:30:28.274843 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.274806 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw"] Nov 03 13:30:28.355231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355198 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.355415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355267 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.355415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355298 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.355415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355343 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.355560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355417 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.355560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355522 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.355653 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355565 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drgz\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-kube-api-access-2drgz\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.355653 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355589 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.355653 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355635 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.355815 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355797 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fae73af-85a0-4129-909d-6be7992d360d-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.355873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355820 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87dqm\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-kube-api-access-87dqm\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.355873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355861 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.355964 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355876 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.356005 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.355978 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.356057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356039 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.356181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356162 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.356240 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356221 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.356305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356253 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.356305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356283 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fae73af-85a0-4129-909d-6be7992d360d-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.356579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356554 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.356667 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356564 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fae73af-85a0-4129-909d-6be7992d360d-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.356667 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356589 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.356825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.356803 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.358701 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.358672 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.359019 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.358998 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fae73af-85a0-4129-909d-6be7992d360d-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.410383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.410342 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drgz\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-kube-api-access-2drgz\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.410536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.410353 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.457124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457026 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.457124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457060 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.457124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457110 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.457124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457128 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.457441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457145 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.457441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457191 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.457441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457351 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87dqm\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-kube-api-access-87dqm\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.457643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457552 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.457933 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457903 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.458015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.457966 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.458015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.458004 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.458181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.458065 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.458181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.458150 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.458330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.458311 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.459542 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.459517 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.459930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.459913 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.485110 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.485052 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:28.511267 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.511240 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.511784 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.511766 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87dqm\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-kube-api-access-87dqm\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.528867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.528210 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:28.905592 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:30:28.905558 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fae73af_85a0_4129_909d_6be7992d360d.slice/crio-fb2f548da46cc8940cdb4407125316aae192f1ee00fc76c221427ab76465d4ad WatchSource:0}: Error finding container fb2f548da46cc8940cdb4407125316aae192f1ee00fc76c221427ab76465d4ad: Status 404 returned error can't find the container with id fb2f548da46cc8940cdb4407125316aae192f1ee00fc76c221427ab76465d4ad Nov 03 13:30:28.905897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.905875 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8"] Nov 03 13:30:28.908178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.908146 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:30:28.908248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.908212 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:30:28.908248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.908243 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:30:28.957665 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:30:28.957634 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70a0384_eb61_46e6_8b0d_0105f34de2e3.slice/crio-333785298e78bb742d17fa391ad23cdf5486e99fc63fc71f58dfb8713e9f7ad4 WatchSource:0}: Error finding container 333785298e78bb742d17fa391ad23cdf5486e99fc63fc71f58dfb8713e9f7ad4: Status 404 returned error can't find the container with id 333785298e78bb742d17fa391ad23cdf5486e99fc63fc71f58dfb8713e9f7ad4 Nov 03 13:30:28.959349 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.959264 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw"] Nov 03 13:30:28.960373 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.960339 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:30:28.960531 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.960506 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:30:28.960624 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:28.960556 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:30:29.105195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.105151 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" event={"ID":"d70a0384-eb61-46e6-8b0d-0105f34de2e3","Type":"ContainerStarted","Data":"afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5"} Nov 03 13:30:29.105390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.105255 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" event={"ID":"d70a0384-eb61-46e6-8b0d-0105f34de2e3","Type":"ContainerStarted","Data":"333785298e78bb742d17fa391ad23cdf5486e99fc63fc71f58dfb8713e9f7ad4"} Nov 03 13:30:29.106949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.106923 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" event={"ID":"2fae73af-85a0-4129-909d-6be7992d360d","Type":"ContainerStarted","Data":"96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e"} Nov 03 13:30:29.107013 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.106967 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" event={"ID":"2fae73af-85a0-4129-909d-6be7992d360d","Type":"ContainerStarted","Data":"fb2f548da46cc8940cdb4407125316aae192f1ee00fc76c221427ab76465d4ad"} Nov 03 13:30:29.222983 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.222918 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" podStartSLOduration=1.222899275 podStartE2EDuration="1.222899275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:29.216489134 +0000 UTC m=+1562.410892728" watchObservedRunningTime="2026-11-03 13:30:29.222899275 +0000 UTC m=+1562.417302871" Nov 03 13:30:29.332557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.332487 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podStartSLOduration=1.332453791 podStartE2EDuration="1.332453791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:29.330291558 +0000 UTC m=+1562.524695154" watchObservedRunningTime="2026-11-03 13:30:29.332453791 +0000 UTC m=+1562.526857386" Nov 03 13:30:29.485436 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.485353 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:29.486785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.486762 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:30:29.486903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.486805 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 13:30:29.529294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:29.529259 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:30.150150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:30.150112 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:30.488466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:30.488419 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:30:30.488878 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:30.488502 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 13:30:30.531705 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:30.531668 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:30:30.531967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:30.531937 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:30:30.752782 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:30.752652 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:31.022417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:31.022288 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:31.208308 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:31.208271 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:31.492458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:31.492255 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:30:31.492458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:31.492316 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 13:30:31.534096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:31.530476 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:30:31.534096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:31.530551 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:30:32.079097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:32.075599 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764"] Nov 03 13:30:32.111276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:32.111236 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1853b0-g2764"] Nov 03 13:30:32.495291 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:32.495253 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:30:32.498097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:32.495863 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 13:30:32.547640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:32.547497 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:30:32.547640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:32.547579 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:30:32.997421 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:32.997036 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ee6910-4a13-4a35-af0f-541c0fd73659" path="/var/lib/kubelet/pods/69ee6910-4a13-4a35-af0f-541c0fd73659/volumes" Nov 03 13:30:33.487603 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:33.487561 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:30:33.487875 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:33.487844 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 13:30:33.530484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:33.530446 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:30:33.531057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:33.531025 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:30:34.485560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:34.485518 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:30:34.485853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:34.485819 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 13:30:34.531374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:34.531334 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:30:34.531968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:34.531932 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:30:35.486483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:35.486432 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:30:35.486658 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:35.486508 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 13:30:35.534096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:35.531446 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:30:35.534096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:35.531513 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:30:36.274688 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.274639 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:36.425687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.425653 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h"] Nov 03 13:30:36.432367 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.432338 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.482156 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.481940 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-461b2d-serving-cert\"" Nov 03 13:30:36.483654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.482829 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h"] Nov 03 13:30:36.483654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.483088 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.483654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.483273 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvmz\" (UniqueName: \"kubernetes.io/projected/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-kube-api-access-hpvmz\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.483654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.483377 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-metrics-certs\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.483654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.483487 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.585145 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.584528 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-metrics-certs\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.585145 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.584637 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.585145 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.584702 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.585145 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.584876 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvmz\" (UniqueName: \"kubernetes.io/projected/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-kube-api-access-hpvmz\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.593290 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.592868 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.594117 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:36.594091 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-serving-cert: secret "maas-api-e2e-api-a-461b2d-serving-cert" not found Nov 03 13:30:36.594351 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:36.594338 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls podName:0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7 nodeName:}" failed. No retries permitted until 2026-11-03 13:30:37.09431761 +0000 UTC m=+1570.288721210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls") pod "maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" (UID: "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7") : secret "maas-api-e2e-api-a-461b2d-serving-cert" not found Nov 03 13:30:36.596849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.596824 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-metrics-certs\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.617162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.612449 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:36.640729 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.640698 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvmz\" (UniqueName: \"kubernetes.io/projected/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-kube-api-access-hpvmz\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:36.775062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.774973 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:36.894121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:36.893859 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:37.066096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.064662 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:37.066096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.066031 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:37.071626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.067252 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 13:30:37.122395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.122356 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:37.125231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.125205 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls\") pod \"maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:37.185448 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.185413 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:37.186035 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.186009 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:37.187273 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.187248 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:30:37.353899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.353863 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:37.751980 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:30:37.751942 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c36e1e6_f31e_4543_87ca_9f96d2d0e6a7.slice/crio-07fa679344c862c2eedda150a7ea3c750087e3cf4b0c791090e4ebe380d4e59a WatchSource:0}: Error finding container 07fa679344c862c2eedda150a7ea3c750087e3cf4b0c791090e4ebe380d4e59a: Status 404 returned error can't find the container with id 07fa679344c862c2eedda150a7ea3c750087e3cf4b0c791090e4ebe380d4e59a Nov 03 13:30:37.753467 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.753439 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h"] Nov 03 13:30:37.753610 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:37.753592 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:30:38.307148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:38.307109 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" event={"ID":"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7","Type":"ContainerStarted","Data":"07fa679344c862c2eedda150a7ea3c750087e3cf4b0c791090e4ebe380d4e59a"} Nov 03 13:30:39.315710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:39.315672 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" event={"ID":"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7","Type":"ContainerStarted","Data":"90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96"} Nov 03 13:30:39.316172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:39.315766 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:39.424653 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:39.424596 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" podStartSLOduration=2.557078832 podStartE2EDuration="3.424581682s" totalImagesPullingTime="867.50285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:39.423653063 +0000 UTC m=+1572.618056656" watchObservedRunningTime="2026-11-03 13:30:39.424581682 +0000 UTC m=+1572.618985275" Nov 03 13:30:41.955733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:41.955693 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:45.327990 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:45.327960 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:30:46.991393 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:46.991358 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:47.162484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:47.162433 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:47.293895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:47.293812 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:48.561131 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:48.561051 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:30:52.339688 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:52.339647 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:56.808313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:56.808273 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r"] Nov 03 13:30:56.808752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:56.808489 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" podUID="621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" containerName="maas-api" containerID="cri-o://b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9" gracePeriod=30 Nov 03 13:30:56.894602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:56.894559 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:57.183763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.183731 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:30:57.306871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.306590 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-openshift-service-ca\") pod \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " Nov 03 13:30:57.306871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.306685 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-kube-api-access-4v9ng\" (UniqueName: \"kubernetes.io/projected/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-kube-api-access-4v9ng\") pod \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " Nov 03 13:30:57.306871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.306721 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-metrics-certs\") pod \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " Nov 03 13:30:57.306871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.306847 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls\") pod \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\" (UID: \"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89\") " Nov 03 13:30:57.310639 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.310608 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-openshift-service-ca" pod "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" (UID: "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:57.311657 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.311613 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls" pod "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" (UID: "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:30:57.331571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.331513 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-metrics-certs" pod "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" (UID: "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:30:57.332180 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.332141 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-kube-api-access-4v9ng" pod "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" (UID: "621679fd-6f97-4deb-bc4e-7fc4e7d2ff89"). InnerVolumeSpecName "kube-api-access-4v9ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:57.410646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.410605 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:57.410646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.410648 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v9ng\" (UniqueName: \"kubernetes.io/projected/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-kube-api-access-4v9ng\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:57.410896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.410667 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:57.410896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.410680 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:57.459162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.459121 4924 generic.go:356] "Generic (PLEG): container finished" podID="621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" containerID="b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9" exitCode=0 Nov 03 13:30:57.459392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.459223 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" event={"ID":"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89","Type":"ContainerDied","Data":"b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9"} Nov 03 13:30:57.459392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.459301 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" event={"ID":"621679fd-6f97-4deb-bc4e-7fc4e7d2ff89","Type":"ContainerDied","Data":"0875d99e99ffd5ce14c25b97899316e65bedcfe2d2cbba321d0df3cbc5312dfa"} Nov 03 13:30:57.459392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.459326 4924 scope.go:122] "RemoveContainer" containerID="b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9" Nov 03 13:30:57.459392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.459237 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r" Nov 03 13:30:57.485896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.485871 4924 scope.go:122] "RemoveContainer" containerID="b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9" Nov 03 13:30:57.486243 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:30:57.486221 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9\": container with ID starting with b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9 not found: ID does not exist" containerID="b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9" Nov 03 13:30:57.486332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.486252 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9"} err="failed to get container status \"b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9\": rpc error: code = NotFound desc = could not find container \"b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9\": container with ID starting with b161b48c02fc6ccaf212e181f2284521bb4f56bb69b05a5bf2161229576fc0b9 not found: ID does not exist" Nov 03 13:30:57.617562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.617531 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r"] Nov 03 13:30:57.622863 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:57.622838 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30074e-6b859b9bcc-lf29r"] Nov 03 13:30:58.209341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.209303 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr"] Nov 03 13:30:58.209844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.209590 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" podUID="ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" containerName="maas-api" containerID="cri-o://2f4c47279ab8849108fc9013da3b416428092273cb456052ad8d61be78a73761" gracePeriod=30 Nov 03 13:30:58.309183 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.309146 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:30:58.517968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.517802 4924 generic.go:356] "Generic (PLEG): container finished" podID="ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" containerID="2f4c47279ab8849108fc9013da3b416428092273cb456052ad8d61be78a73761" exitCode=0 Nov 03 13:30:58.517968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.517942 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" event={"ID":"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624","Type":"ContainerDied","Data":"2f4c47279ab8849108fc9013da3b416428092273cb456052ad8d61be78a73761"} Nov 03 13:30:58.593004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.592974 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:30:58.732159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.729202 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-kube-api-access-7tshs\" (UniqueName: \"kubernetes.io/projected/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-kube-api-access-7tshs\") pod \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " Nov 03 13:30:58.732159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.729324 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-metrics-certs\") pod \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " Nov 03 13:30:58.732159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.729408 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-openshift-service-ca\") pod \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " Nov 03 13:30:58.732159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.729465 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls\") pod \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\" (UID: \"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624\") " Nov 03 13:30:58.734799 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.732691 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls" pod "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" (UID: "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:30:58.734799 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.733984 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-openshift-service-ca" pod "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" (UID: "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:58.737849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.736476 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-kube-api-access-7tshs" pod "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" (UID: "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624"). InnerVolumeSpecName "kube-api-access-7tshs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:58.738651 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.738610 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-metrics-certs" pod "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" (UID: "ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:30:58.830858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.830828 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:58.830858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.830855 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:58.831118 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.830865 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:58.831118 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.830874 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tshs\" (UniqueName: \"kubernetes.io/projected/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624-kube-api-access-7tshs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:58.962263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:58.962218 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" path="/var/lib/kubelet/pods/621679fd-6f97-4deb-bc4e-7fc4e7d2ff89/volumes" Nov 03 13:30:59.527864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:59.527826 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" Nov 03 13:30:59.527864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:59.527855 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr" event={"ID":"ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624","Type":"ContainerDied","Data":"7a0e366f1c4ce65bdf15812d2844b086104ed9cfe867d95e3d8ea59501a07fea"} Nov 03 13:30:59.528466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:59.527918 4924 scope.go:122] "RemoveContainer" containerID="2f4c47279ab8849108fc9013da3b416428092273cb456052ad8d61be78a73761" Nov 03 13:30:59.687402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:59.687365 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr"] Nov 03 13:30:59.734528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:30:59.734484 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-jb8hr"] Nov 03 13:31:00.021258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:00.021227 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:00.369528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:00.369439 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:00.508578 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:00.508539 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:00.954595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:00.954557 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" path="/var/lib/kubelet/pods/ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624/volumes" Nov 03 13:31:01.232709 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:01.232614 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:01.375040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:01.374996 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:02.742631 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:02.742591 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:03.325136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:03.322482 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:04.105558 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:04.105523 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:04.705185 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:04.705149 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:06.268669 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:06.268635 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:08.700564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:08.700528 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:08.934734 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:08.934692 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:10.551732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:10.551694 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:11.264730 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.264692 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67"] Nov 03 13:31:11.264914 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.264892 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" podUID="7369abf4-255b-4b8f-8393-61973d08a0df" containerName="maas-api" containerID="cri-o://8327511c85da0e96f1ca8aa8e1a049edccaf2af070466c4bf188d905a6b14ccb" gracePeriod=30 Nov 03 13:31:11.280968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.280932 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:11.451582 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.451541 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:11.633213 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.628664 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:11.659030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.658992 4924 generic.go:356] "Generic (PLEG): container finished" podID="7369abf4-255b-4b8f-8393-61973d08a0df" containerID="8327511c85da0e96f1ca8aa8e1a049edccaf2af070466c4bf188d905a6b14ccb" exitCode=0 Nov 03 13:31:11.660670 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.660637 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" event={"ID":"7369abf4-255b-4b8f-8393-61973d08a0df","Type":"ContainerDied","Data":"8327511c85da0e96f1ca8aa8e1a049edccaf2af070466c4bf188d905a6b14ccb"} Nov 03 13:31:11.661052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.661025 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" event={"ID":"7369abf4-255b-4b8f-8393-61973d08a0df","Type":"ContainerDied","Data":"91803b669643138b650dd4abc985e1a61f95970de8f2aff90d3b4703254ab7d5"} Nov 03 13:31:11.661281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.661263 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91803b669643138b650dd4abc985e1a61f95970de8f2aff90d3b4703254ab7d5" Nov 03 13:31:11.675411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.675381 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:31:11.745870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.745745 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7369abf4-255b-4b8f-8393-61973d08a0df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369abf4-255b-4b8f-8393-61973d08a0df-openshift-service-ca\") pod \"7369abf4-255b-4b8f-8393-61973d08a0df\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " Nov 03 13:31:11.746606 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.746579 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls\") pod \"7369abf4-255b-4b8f-8393-61973d08a0df\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " Nov 03 13:31:11.746808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.746792 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-metrics-certs\") pod \"7369abf4-255b-4b8f-8393-61973d08a0df\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " Nov 03 13:31:11.747389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.747362 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7369abf4-255b-4b8f-8393-61973d08a0df-kube-api-access-vppsm\" (UniqueName: \"kubernetes.io/projected/7369abf4-255b-4b8f-8393-61973d08a0df-kube-api-access-vppsm\") pod \"7369abf4-255b-4b8f-8393-61973d08a0df\" (UID: \"7369abf4-255b-4b8f-8393-61973d08a0df\") " Nov 03 13:31:11.758144 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.747516 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7369abf4-255b-4b8f-8393-61973d08a0df-openshift-service-ca" pod "7369abf4-255b-4b8f-8393-61973d08a0df" (UID: "7369abf4-255b-4b8f-8393-61973d08a0df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:11.792648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.792318 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-metrics-certs" pod "7369abf4-255b-4b8f-8393-61973d08a0df" (UID: "7369abf4-255b-4b8f-8393-61973d08a0df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:31:11.792648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.792329 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7369abf4-255b-4b8f-8393-61973d08a0df-kube-api-access-vppsm" pod "7369abf4-255b-4b8f-8393-61973d08a0df" (UID: "7369abf4-255b-4b8f-8393-61973d08a0df"). InnerVolumeSpecName "kube-api-access-vppsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:11.794384 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.794338 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls" pod "7369abf4-255b-4b8f-8393-61973d08a0df" (UID: "7369abf4-255b-4b8f-8393-61973d08a0df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:31:11.861017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.860922 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.861017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.860983 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7369abf4-255b-4b8f-8393-61973d08a0df-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.861017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.860999 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppsm\" (UniqueName: \"kubernetes.io/projected/7369abf4-255b-4b8f-8393-61973d08a0df-kube-api-access-vppsm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.861017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.861015 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7369abf4-255b-4b8f-8393-61973d08a0df-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.962281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:11.962200 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:12.166848 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:12.166814 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:12.669430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:12.669398 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67" Nov 03 13:31:12.826762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:12.826665 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67"] Nov 03 13:31:12.873122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:12.873091 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-abe0e0-56c767dfb8-nwm67"] Nov 03 13:31:12.951714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:12.951621 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7369abf4-255b-4b8f-8393-61973d08a0df" path="/var/lib/kubelet/pods/7369abf4-255b-4b8f-8393-61973d08a0df/volumes" Nov 03 13:31:13.577079 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:13.577030 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:16.076608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:16.076572 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:17.107909 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:17.107852 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:18.559740 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:31:18.559703 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:31:19.143630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:19.143589 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:21.766898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:21.766854 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:21.896428 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:21.896385 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:23.397294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:23.397253 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:23.983188 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:23.983142 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:25.171987 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:25.171955 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:25.693571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:25.693532 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:30.912235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:30.912199 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:32.436406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.436369 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v"] Nov 03 13:31:32.437042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437023 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" containerName="maas-api" Nov 03 13:31:32.437121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437043 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" containerName="maas-api" Nov 03 13:31:32.437121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437066 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7369abf4-255b-4b8f-8393-61973d08a0df" containerName="maas-api" Nov 03 13:31:32.437121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437082 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7369abf4-255b-4b8f-8393-61973d08a0df" containerName="maas-api" Nov 03 13:31:32.437121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437099 4924 cpu_manager.go:414] "containerMap: removing container" podUID="621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" containerName="maas-api" Nov 03 13:31:32.437121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437104 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" containerName="maas-api" Nov 03 13:31:32.437268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437203 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7369abf4-255b-4b8f-8393-61973d08a0df" containerName="maas-api" Nov 03 13:31:32.437268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437213 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="621679fd-6f97-4deb-bc4e-7fc4e7d2ff89" containerName="maas-api" Nov 03 13:31:32.437268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.437222 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2fe15a-4cf3-48c4-a8e4-46ca1f2f6624" containerName="maas-api" Nov 03 13:31:32.442179 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.442158 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.482484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.482449 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.482646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.482500 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.482646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.482616 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2mk\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-kube-api-access-bn2mk\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.482726 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.482662 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.482792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.482770 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.482972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.482954 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.483009 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.482995 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.483049 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.483032 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.483129 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.483113 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.491525 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.491500 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-28d87c-openshift-default-dockercfg-lcrzm\"" Nov 03 13:31:32.543394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.543350 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v"] Nov 03 13:31:32.584475 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584434 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.584664 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584535 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.584664 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584570 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.584664 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584655 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2mk\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-kube-api-access-bn2mk\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.584849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584712 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.584849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584756 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.584953 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584875 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.584953 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584936 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.585058 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.584973 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.585058 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.585005 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.585373 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.585340 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.585604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.585567 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.585712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.585654 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.586017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.585993 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.587779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.587756 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.588202 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.588183 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.639375 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.639335 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.639617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.639597 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2mk\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-kube-api-access-bn2mk\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:32.757652 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:32.757551 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:33.180594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:33.180566 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v"] Nov 03 13:31:33.181234 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:31:33.181205 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8067e6f_06cc_48c5_8e3e_af27a06a8f87.slice/crio-17b68e0aa110882792ea092649b87d26c69a8fd1c93a159b59e6df691fd7f794 WatchSource:0}: Error finding container 17b68e0aa110882792ea092649b87d26c69a8fd1c93a159b59e6df691fd7f794: Status 404 returned error can't find the container with id 17b68e0aa110882792ea092649b87d26c69a8fd1c93a159b59e6df691fd7f794 Nov 03 13:31:33.183786 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:33.183754 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:33.183860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:33.183823 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:33.183894 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:33.183873 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:33.858681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:33.858643 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" event={"ID":"e8067e6f-06cc-48c5-8e3e-af27a06a8f87","Type":"ContainerStarted","Data":"aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40"} Nov 03 13:31:33.859209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:33.858734 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" event={"ID":"e8067e6f-06cc-48c5-8e3e-af27a06a8f87","Type":"ContainerStarted","Data":"17b68e0aa110882792ea092649b87d26c69a8fd1c93a159b59e6df691fd7f794"} Nov 03 13:31:34.061097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:34.061000 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" podStartSLOduration=2.060986556 podStartE2EDuration="2.060986556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:34.01418779 +0000 UTC m=+1627.208591384" watchObservedRunningTime="2026-11-03 13:31:34.060986556 +0000 UTC m=+1627.255390149" Nov 03 13:31:34.758010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:34.757975 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:34.759894 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:34.759865 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 13:31:34.760021 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:34.759918 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 13:31:35.758665 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:35.758618 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 13:31:35.759218 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:35.758683 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 13:31:36.758607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:36.758572 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 13:31:36.758813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:36.758632 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 13:31:36.993043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:36.992998 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:37.612380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:37.612347 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk"] Nov 03 13:31:37.612559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:37.612538 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" containerID="cri-o://9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591" gracePeriod=30 Nov 03 13:31:37.659844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:37.659808 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797"] Nov 03 13:31:37.660106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:37.660056 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" containerID="cri-o://506ae1324d2a4f11d400c7eea3226d5d24e4af97cdd9facb0c7f43f1a7568586" gracePeriod=30 Nov 03 13:31:37.759542 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:37.759506 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 13:31:37.760192 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:37.760160 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 13:31:38.759148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:38.759102 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 13:31:38.759340 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:38.759197 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 13:31:38.767050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:38.767011 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w"] Nov 03 13:31:38.767504 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:38.767302 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" containerID="cri-o://faf301a4c1b548744caa5f744497869fd9074b5820d446353ee728a34de25f90" gracePeriod=30 Nov 03 13:31:39.762895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:39.762864 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:39.763204 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:39.763182 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:39.763830 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:39.763811 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:31:42.089031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.088996 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:31:42.089535 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.089091 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:31:42.226331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.226296 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:42.949026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.948990 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:31:42.952121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.951881 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:42.976576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.976545 4924 generic.go:356] "Generic (PLEG): container finished" podID="846adfab-75a2-4825-af40-05bc343109ae" containerID="506ae1324d2a4f11d400c7eea3226d5d24e4af97cdd9facb0c7f43f1a7568586" exitCode=0 Nov 03 13:31:42.976717 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.976614 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" event={"ID":"846adfab-75a2-4825-af40-05bc343109ae","Type":"ContainerDied","Data":"506ae1324d2a4f11d400c7eea3226d5d24e4af97cdd9facb0c7f43f1a7568586"} Nov 03 13:31:42.978143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.978110 4924 generic.go:356] "Generic (PLEG): container finished" podID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerID="9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591" exitCode=0 Nov 03 13:31:42.978245 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.978192 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" event={"ID":"6733a046-e91a-4430-9d6c-5756d05b91f1","Type":"ContainerDied","Data":"9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591"} Nov 03 13:31:42.978245 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.978207 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" Nov 03 13:31:42.978245 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.978235 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk" event={"ID":"6733a046-e91a-4430-9d6c-5756d05b91f1","Type":"ContainerDied","Data":"08ee2db3cd524e8ed6de02115d86e2998ea4192ca7e8c9b077b1bfc8271623bc"} Nov 03 13:31:42.978361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.978275 4924 scope.go:122] "RemoveContainer" containerID="9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591" Nov 03 13:31:42.991536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.991519 4924 scope.go:122] "RemoveContainer" containerID="9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591" Nov 03 13:31:42.991800 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:31:42.991780 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591\": container with ID starting with 9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591 not found: ID does not exist" containerID="9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591" Nov 03 13:31:42.991877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.991814 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591"} err="failed to get container status \"9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591\": rpc error: code = NotFound desc = could not find container \"9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591\": container with ID starting with 9ff63bc81137cb1acbd37bbe6a794a9c4fa7b469c80b03a80b3a108c67582591 not found: ID does not exist" Nov 03 13:31:42.993633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:42.993618 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:31:43.050555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050518 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-podinfo\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.050741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050606 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-credential-socket\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.050741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050642 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-token\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-token\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.050741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050695 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6733a046-e91a-4430-9d6c-5756d05b91f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733a046-e91a-4430-9d6c-5756d05b91f1-istiod-ca-cert\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.050741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050719 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-certs\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.050950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050757 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-envoy\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.050950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050825 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-istio-token\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-istio-token\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.050950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050857 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-envoy\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.050950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050901 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-kube-api-access-rrz65\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-kube-api-access-rrz65\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.050950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.050900 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-credential-socket" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.051249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051002 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-socket\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.051249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051021 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-certs" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.051249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051055 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-certs\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.051249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051147 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-credential-socket\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.051249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051153 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6733a046-e91a-4430-9d6c-5756d05b91f1-istiod-ca-cert" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:43.051249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051227 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-socket\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.051553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051260 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846adfab-75a2-4825-af40-05bc343109ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846adfab-75a2-4825-af40-05bc343109ae-istiod-ca-cert\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.051553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051289 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-data\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.051553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051380 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/846adfab-75a2-4825-af40-05bc343109ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846adfab-75a2-4825-af40-05bc343109ae-istio-podinfo\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.051553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051433 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-data\") pod \"6733a046-e91a-4430-9d6c-5756d05b91f1\" (UID: \"6733a046-e91a-4430-9d6c-5756d05b91f1\") " Nov 03 13:31:43.051553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051456 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-socket" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.051553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051503 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-kube-api-access-7qtkr\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-kube-api-access-7qtkr\") pod \"846adfab-75a2-4825-af40-05bc343109ae\" (UID: \"846adfab-75a2-4825-af40-05bc343109ae\") " Nov 03 13:31:43.051869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.051655 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-certs" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.052286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052263 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-data" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.052286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052281 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.052459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052300 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733a046-e91a-4430-9d6c-5756d05b91f1-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.052459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052315 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.052459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052329 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.052459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052344 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.053264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052645 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846adfab-75a2-4825-af40-05bc343109ae-istiod-ca-cert" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:43.053264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052795 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-socket" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.053264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.052857 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-credential-socket" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.053264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.053147 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-data" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.053557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.053530 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-envoy" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.053917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.053875 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-podinfo" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:43.054040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.053982 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-token" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.054040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.054017 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-kube-api-access-7qtkr" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "kube-api-access-7qtkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.054201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.054128 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-kube-api-access-rrz65" pod "6733a046-e91a-4430-9d6c-5756d05b91f1" (UID: "6733a046-e91a-4430-9d6c-5756d05b91f1"). InnerVolumeSpecName "kube-api-access-rrz65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.054703 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.054674 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-envoy" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.054786 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.054769 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-istio-token" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.054965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.054946 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/846adfab-75a2-4825-af40-05bc343109ae-istio-podinfo" pod "846adfab-75a2-4825-af40-05bc343109ae" (UID: "846adfab-75a2-4825-af40-05bc343109ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:43.154385 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154343 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154385 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154386 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154403 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846adfab-75a2-4825-af40-05bc343109ae-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154417 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154438 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846adfab-75a2-4825-af40-05bc343109ae-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154452 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154464 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtkr\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-kube-api-access-7qtkr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154481 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154494 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154508 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846adfab-75a2-4825-af40-05bc343109ae-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154521 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846adfab-75a2-4825-af40-05bc343109ae-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154535 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733a046-e91a-4430-9d6c-5756d05b91f1-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.154957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.154549 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrz65\" (UniqueName: \"kubernetes.io/projected/6733a046-e91a-4430-9d6c-5756d05b91f1-kube-api-access-rrz65\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.164970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.164821 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:43.400558 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.400521 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk"] Nov 03 13:31:43.404534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.404502 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hdtwk"] Nov 03 13:31:43.611343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.611246 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:31:43.611343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.611316 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:31:43.666147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.664330 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:43.887840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:43.886418 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:44.022516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.020848 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:44.029089 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:31:44.024433 4924 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b974cad_c05f_4cd4_a8e5_a73a4acfd506.slice/crio-conmon-faf301a4c1b548744caa5f744497869fd9074b5820d446353ee728a34de25f90.scope\": RecentStats: unable to find data in memory cache]" Nov 03 13:31:44.032548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.032382 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" Nov 03 13:31:44.032548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.032399 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797" event={"ID":"846adfab-75a2-4825-af40-05bc343109ae","Type":"ContainerDied","Data":"bd4f61bb7f1db48eabd1752e712d897ccb3ad88289e52e8c84178f43931be7a8"} Nov 03 13:31:44.032548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.032466 4924 scope.go:122] "RemoveContainer" containerID="506ae1324d2a4f11d400c7eea3226d5d24e4af97cdd9facb0c7f43f1a7568586" Nov 03 13:31:44.035417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.035385 4924 generic.go:356] "Generic (PLEG): container finished" podID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerID="faf301a4c1b548744caa5f744497869fd9074b5820d446353ee728a34de25f90" exitCode=0 Nov 03 13:31:44.035526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.035473 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" event={"ID":"1b974cad-c05f-4cd4-a8e5-a73a4acfd506","Type":"ContainerDied","Data":"faf301a4c1b548744caa5f744497869fd9074b5820d446353ee728a34de25f90"} Nov 03 13:31:44.214846 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.214815 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797"] Nov 03 13:31:44.216632 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.216610 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:31:44.219831 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.219789 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-2m797"] Nov 03 13:31:44.391019 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.390986 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-socket\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391190 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391116 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-envoy\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391190 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391175 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-data\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391233 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-podinfo\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391259 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-credential-socket\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391345 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-kube-api-access-ss26f\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-kube-api-access-ss26f\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391343 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-socket" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.391446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391380 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-certs\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391428 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istiod-ca-cert\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391470 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-token\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-token\") pod \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\" (UID: \"1b974cad-c05f-4cd4-a8e5-a73a4acfd506\") " Nov 03 13:31:44.391648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391583 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-data" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.391917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391893 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istiod-ca-cert" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:44.392057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.391736 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-certs" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.392057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.392047 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.392211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.392066 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.392211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.392095 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.392211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.392091 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-credential-socket" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.393867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.393831 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-envoy" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.394100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.394043 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-token" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.394751 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.394729 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-kube-api-access-ss26f" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "kube-api-access-ss26f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.394838 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.394790 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-podinfo" pod "1b974cad-c05f-4cd4-a8e5-a73a4acfd506" (UID: "1b974cad-c05f-4cd4-a8e5-a73a4acfd506"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:44.493950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.493868 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.493950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.493902 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.493950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.493914 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.493950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.493924 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.493950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.493933 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss26f\" (UniqueName: \"kubernetes.io/projected/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-kube-api-access-ss26f\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.493950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.493942 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b974cad-c05f-4cd4-a8e5-a73a4acfd506-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.957040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.956998 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" path="/var/lib/kubelet/pods/6733a046-e91a-4430-9d6c-5756d05b91f1/volumes" Nov 03 13:31:44.957747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:44.957723 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846adfab-75a2-4825-af40-05bc343109ae" path="/var/lib/kubelet/pods/846adfab-75a2-4825-af40-05bc343109ae/volumes" Nov 03 13:31:45.049591 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:45.049557 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" Nov 03 13:31:45.049591 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:45.049579 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w" event={"ID":"1b974cad-c05f-4cd4-a8e5-a73a4acfd506","Type":"ContainerDied","Data":"8f3188e3260746b14e896a68cd4b0856671d81ab829ec88e477b4a8358918f75"} Nov 03 13:31:45.049851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:45.049651 4924 scope.go:122] "RemoveContainer" containerID="faf301a4c1b548744caa5f744497869fd9074b5820d446353ee728a34de25f90" Nov 03 13:31:45.256259 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:45.256153 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w"] Nov 03 13:31:45.300715 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:45.300684 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-c6k8w"] Nov 03 13:31:45.606736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:45.606646 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:45.862160 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:45.861407 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:46.952434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:46.952402 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" path="/var/lib/kubelet/pods/1b974cad-c05f-4cd4-a8e5-a73a4acfd506/volumes" Nov 03 13:31:48.559964 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:31:48.559924 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:31:48.767776 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:48.767741 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:49.241451 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:49.241413 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:49.574568 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:49.574487 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:49.743001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:49.742972 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:52.088095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:52.087632 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:52.286093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:52.285618 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:52.503998 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:52.503794 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:52.916211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:52.916179 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:53.764096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:53.763227 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:31:53.764096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:53.763345 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:53.914217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:53.914173 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:31:53.914567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:53.914533 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:54.073840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.073750 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 03 13:31:54.075153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.075120 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:54.104050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.103172 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" podUID="077a2a59-e1a7-4f11-af5d-4373cc6643ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:54.421095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.409163 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" podUID="723ac416-02d4-48a5-a50b-21fb78a3fce5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Nov 03 13:31:54.511095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.505149 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" podUID="9542ec83-268f-48d4-a677-4b42c3ac9c5f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 13:31:54.511095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.506497 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:54.640554 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.640507 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s"] Nov 03 13:31:54.680867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.680344 4924 cpu_manager.go:414] "containerMap: removing container" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" Nov 03 13:31:54.680867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.680377 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" Nov 03 13:31:54.680867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.680406 4924 cpu_manager.go:414] "containerMap: removing container" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" Nov 03 13:31:54.680867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.680414 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" Nov 03 13:31:54.680867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.680437 4924 cpu_manager.go:414] "containerMap: removing container" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" Nov 03 13:31:54.680867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.680445 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" Nov 03 13:31:54.682176 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.682151 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="846adfab-75a2-4825-af40-05bc343109ae" containerName="istio-proxy" Nov 03 13:31:54.682355 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.682340 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6733a046-e91a-4430-9d6c-5756d05b91f1" containerName="istio-proxy" Nov 03 13:31:54.682540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.682477 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b974cad-c05f-4cd4-a8e5-a73a4acfd506" containerName="istio-proxy" Nov 03 13:31:54.697539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.697510 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.704164 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.704119 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:54.716141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.716040 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s"] Nov 03 13:31:54.852129 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.848506 4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" podUID="a983cf0a-e87a-4615-a048-df14fb6c2f45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:54.863127 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.860949 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5295ab29-serving-cert\"" Nov 03 13:31:54.864890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.864039 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2935bad-4728-4a81-8a5a-1bb52e55099d-openshift-service-ca\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.864890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.864114 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-maas-api-tls\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.864890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.864358 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-metrics-certs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.864890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.864418 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtls2\" (UniqueName: \"kubernetes.io/projected/a2935bad-4728-4a81-8a5a-1bb52e55099d-kube-api-access-dtls2\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.878674 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.878639 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:54.883523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.883488 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" podUID="a983cf0a-e87a-4615-a048-df14fb6c2f45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:54.972639 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.972535 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-metrics-certs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.973463 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.973435 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtls2\" (UniqueName: \"kubernetes.io/projected/a2935bad-4728-4a81-8a5a-1bb52e55099d-kube-api-access-dtls2\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.974031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.974009 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2935bad-4728-4a81-8a5a-1bb52e55099d-openshift-service-ca\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.975050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.975031 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-maas-api-tls\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.975633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.974948 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2935bad-4728-4a81-8a5a-1bb52e55099d-openshift-service-ca\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.993093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.985627 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-maas-api-tls\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:54.997242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:54.997178 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-metrics-certs\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:55.028094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:55.027634 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:55.053112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:55.050041 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtls2\" (UniqueName: \"kubernetes.io/projected/a2935bad-4728-4a81-8a5a-1bb52e55099d-kube-api-access-dtls2\") pod \"maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:55.370155 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:55.365592 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:55.890740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:55.890710 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s"] Nov 03 13:31:55.891794 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:31:55.891769 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2935bad_4728_4a81_8a5a_1bb52e55099d.slice/crio-11366c4dc99c06fbc39d1e05ced0594cc547b92bb6a2bbd66bff3e36a66ab027 WatchSource:0}: Error finding container 11366c4dc99c06fbc39d1e05ced0594cc547b92bb6a2bbd66bff3e36a66ab027: Status 404 returned error can't find the container with id 11366c4dc99c06fbc39d1e05ced0594cc547b92bb6a2bbd66bff3e36a66ab027 Nov 03 13:31:56.662243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:56.661801 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" event={"ID":"a2935bad-4728-4a81-8a5a-1bb52e55099d","Type":"ContainerStarted","Data":"11366c4dc99c06fbc39d1e05ced0594cc547b92bb6a2bbd66bff3e36a66ab027"} Nov 03 13:31:57.183694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:57.183613 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:57.670228 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:57.670190 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" event={"ID":"a2935bad-4728-4a81-8a5a-1bb52e55099d","Type":"ContainerStarted","Data":"004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a"} Nov 03 13:31:57.670434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:57.670378 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:31:57.782536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:57.782479 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" podStartSLOduration=2.745238341 podStartE2EDuration="3.782465718s" totalImagesPullingTime="1.037227377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:57.781317968 +0000 UTC m=+1650.975721562" watchObservedRunningTime="2026-11-03 13:31:57.782465718 +0000 UTC m=+1650.976869311" Nov 03 13:31:59.237693 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:59.237659 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:59.423874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:59.423839 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:59.567749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:59.567646 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:31:59.838291 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:31:59.838057 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:00.948280 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:00.948243 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:02.697056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.697020 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c"] Nov 03 13:32:02.701499 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.701480 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.768815 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.768783 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c58e2-4740-418d-8bdc-16ed0155d895-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.769010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.768831 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.769010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.768915 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.769010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.768955 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.769010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.768976 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.769221 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.769139 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c58e2-4740-418d-8bdc-16ed0155d895-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.769261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.769244 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5bz\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-kube-api-access-hp5bz\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.769306 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.769292 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.769446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.769426 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.805874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.805834 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c"] Nov 03 13:32:02.870540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870502 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.870741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870561 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c58e2-4740-418d-8bdc-16ed0155d895-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.870741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870644 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5bz\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-kube-api-access-hp5bz\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.870741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870682 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.870903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870771 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.870903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870816 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c58e2-4740-418d-8bdc-16ed0155d895-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.870903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870852 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.870903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870890 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.871166 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870934 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.871166 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.870941 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.871390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.871366 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.871544 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.871401 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.871617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.871459 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.883885 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.883839 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c58e2-4740-418d-8bdc-16ed0155d895-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.884062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.884007 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c58e2-4740-418d-8bdc-16ed0155d895-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.884743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.884717 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.926319 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.926279 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:02.926479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:02.926402 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5bz\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-kube-api-access-hp5bz\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-7426c\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:03.016001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.015913 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:03.446903 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:32:03.446864 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956c58e2_4740_418d_8bdc_16ed0155d895.slice/crio-48c00c0b59818b1ac7e696ab55072d973d765699744fb1d6794d213fe2a5a32b WatchSource:0}: Error finding container 48c00c0b59818b1ac7e696ab55072d973d765699744fb1d6794d213fe2a5a32b: Status 404 returned error can't find the container with id 48c00c0b59818b1ac7e696ab55072d973d765699744fb1d6794d213fe2a5a32b Nov 03 13:32:03.447108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.447087 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c"] Nov 03 13:32:03.449736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.449700 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:03.449849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.449764 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:03.449849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.449795 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:03.584750 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.584715 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:03.698408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.697275 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:32:03.731040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.730997 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:03.915724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.915649 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" event={"ID":"956c58e2-4740-418d-8bdc-16ed0155d895","Type":"ContainerStarted","Data":"0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8"} Nov 03 13:32:03.915918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:03.915733 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" event={"ID":"956c58e2-4740-418d-8bdc-16ed0155d895","Type":"ContainerStarted","Data":"48c00c0b59818b1ac7e696ab55072d973d765699744fb1d6794d213fe2a5a32b"} Nov 03 13:32:04.021172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:04.018976 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:04.021172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:04.020709 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-7426c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:32:04.021172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:04.020769 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:32:04.028018 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:04.027967 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" podStartSLOduration=2.027953024 podStartE2EDuration="2.027953024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:04.02677764 +0000 UTC m=+1657.221181263" watchObservedRunningTime="2026-11-03 13:32:04.027953024 +0000 UTC m=+1657.222356620" Nov 03 13:32:04.339397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:04.339309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:05.018176 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:05.018132 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-7426c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:32:05.018848 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:05.018197 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:32:06.016360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:06.016319 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-7426c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:32:06.016572 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:06.016396 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:32:07.017203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:07.017120 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-7426c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:32:07.017203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:07.017171 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:32:07.069687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:07.069250 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:08.016439 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:08.016404 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-7426c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:32:08.016618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:08.016465 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:32:09.016532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:09.016497 4924 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-7426c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:32:09.016940 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:09.016547 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:32:10.098559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:10.098525 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:10.099699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:10.099675 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:10.100617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:10.100571 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:32:10.605046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:10.605003 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:12.473808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:12.473779 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:14.573184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:14.573140 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:14.781060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:14.780973 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:16.898619 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:16.898582 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb"] Nov 03 13:32:16.899136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:16.898897 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" podUID="4b8aaf08-9c83-4051-ae05-48c4eae0b468" containerName="maas-api" containerID="cri-o://85891d052f3b085014e9350d713439414397225504f69937141a252e24d699a8" gracePeriod=30 Nov 03 13:32:16.989318 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:16.989282 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:17.092049 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.091981 4924 generic.go:356] "Generic (PLEG): container finished" podID="4b8aaf08-9c83-4051-ae05-48c4eae0b468" containerID="85891d052f3b085014e9350d713439414397225504f69937141a252e24d699a8" exitCode=0 Nov 03 13:32:17.092049 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.092002 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" event={"ID":"4b8aaf08-9c83-4051-ae05-48c4eae0b468","Type":"ContainerDied","Data":"85891d052f3b085014e9350d713439414397225504f69937141a252e24d699a8"} Nov 03 13:32:17.337091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.336763 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:32:17.457100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.451585 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls\") pod \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " Nov 03 13:32:17.457100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.451633 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-metrics-certs\") pod \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " Nov 03 13:32:17.457100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.451769 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8aaf08-9c83-4051-ae05-48c4eae0b468-kube-api-access-xdg74\" (UniqueName: \"kubernetes.io/projected/4b8aaf08-9c83-4051-ae05-48c4eae0b468-kube-api-access-xdg74\") pod \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " Nov 03 13:32:17.457100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.451905 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8aaf08-9c83-4051-ae05-48c4eae0b468-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8aaf08-9c83-4051-ae05-48c4eae0b468-openshift-service-ca\") pod \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\" (UID: \"4b8aaf08-9c83-4051-ae05-48c4eae0b468\") " Nov 03 13:32:17.457100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.452968 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8aaf08-9c83-4051-ae05-48c4eae0b468-openshift-service-ca" pod "4b8aaf08-9c83-4051-ae05-48c4eae0b468" (UID: "4b8aaf08-9c83-4051-ae05-48c4eae0b468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:17.462091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.457797 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-metrics-certs" pod "4b8aaf08-9c83-4051-ae05-48c4eae0b468" (UID: "4b8aaf08-9c83-4051-ae05-48c4eae0b468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:32:17.473097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.468898 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls" pod "4b8aaf08-9c83-4051-ae05-48c4eae0b468" (UID: "4b8aaf08-9c83-4051-ae05-48c4eae0b468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:32:17.479261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.474999 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8aaf08-9c83-4051-ae05-48c4eae0b468-kube-api-access-xdg74" pod "4b8aaf08-9c83-4051-ae05-48c4eae0b468" (UID: "4b8aaf08-9c83-4051-ae05-48c4eae0b468"). InnerVolumeSpecName "kube-api-access-xdg74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:17.554059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.553960 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdg74\" (UniqueName: \"kubernetes.io/projected/4b8aaf08-9c83-4051-ae05-48c4eae0b468-kube-api-access-xdg74\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:17.554059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.553993 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8aaf08-9c83-4051-ae05-48c4eae0b468-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:17.554059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.554004 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:17.554059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:17.554014 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8aaf08-9c83-4051-ae05-48c4eae0b468-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:18.101046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:18.101011 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" event={"ID":"4b8aaf08-9c83-4051-ae05-48c4eae0b468","Type":"ContainerDied","Data":"194d640d6faad9831e5f4f60dea0240fd32395b53ae8d8dea50333ba23f2aa65"} Nov 03 13:32:18.101514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:18.101049 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb" Nov 03 13:32:18.101514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:18.101095 4924 scope.go:122] "RemoveContainer" containerID="85891d052f3b085014e9350d713439414397225504f69937141a252e24d699a8" Nov 03 13:32:18.256857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:18.256816 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb"] Nov 03 13:32:18.303282 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:18.303250 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fdd09-59db8bd8f6-kqqxb"] Nov 03 13:32:18.560704 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:32:18.560659 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:32:18.785589 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:18.785558 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:18.951951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:18.951907 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8aaf08-9c83-4051-ae05-48c4eae0b468" path="/var/lib/kubelet/pods/4b8aaf08-9c83-4051-ae05-48c4eae0b468/volumes" Nov 03 13:32:23.902536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:23.902484 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:25.618456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:25.618410 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:25.760615 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:25.760573 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:26.035750 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:26.035718 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:27.293534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:27.293495 4924 scope.go:122] "RemoveContainer" containerID="9ddd56b16e275433dbbeb46620441202eeddd422b473aba6582ab95f0390ef34" Nov 03 13:32:27.321297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:27.321267 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:27.912146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:27.912107 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:28.712629 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:28.712581 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:29.494858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:29.494800 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:34.945603 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:34.945566 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:35.182107 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:35.182061 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:35.861473 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:35.861434 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:35.990499 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:35.990460 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:39.989214 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:39.989181 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn"] Nov 03 13:32:39.989753 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:39.989739 4924 cpu_manager.go:414] "containerMap: removing container" podUID="4b8aaf08-9c83-4051-ae05-48c4eae0b468" containerName="maas-api" Nov 03 13:32:39.989796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:39.989753 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8aaf08-9c83-4051-ae05-48c4eae0b468" containerName="maas-api" Nov 03 13:32:39.989858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:39.989849 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8aaf08-9c83-4051-ae05-48c4eae0b468" containerName="maas-api" Nov 03 13:32:39.993391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:39.993373 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.045444 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.045410 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e7ca74-serving-cert\"" Nov 03 13:32:40.078113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.074916 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-metrics-certs\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.078113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.075051 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.078113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.075202 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddd6f5e-23e4-4778-909b-79a2ff540375-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.078113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.075339 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2xn\" (UniqueName: \"kubernetes.io/projected/2ddd6f5e-23e4-4778-909b-79a2ff540375-kube-api-access-4z2xn\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.095839 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.095808 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn"] Nov 03 13:32:40.165103 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.165060 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:40.176637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.176609 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2xn\" (UniqueName: \"kubernetes.io/projected/2ddd6f5e-23e4-4778-909b-79a2ff540375-kube-api-access-4z2xn\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.176810 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.176691 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-metrics-certs\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.176810 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.176738 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.176943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.176809 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddd6f5e-23e4-4778-909b-79a2ff540375-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.176943 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:32:40.176913 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-serving-cert: secret "maas-api-e2e-api-b-e7ca74-serving-cert" not found Nov 03 13:32:40.177058 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:32:40.176985 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls podName:2ddd6f5e-23e4-4778-909b-79a2ff540375 nodeName:}" failed. No retries permitted until 2026-11-03 13:32:40.676963068 +0000 UTC m=+1693.871366655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls") pod "maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" (UID: "2ddd6f5e-23e4-4778-909b-79a2ff540375") : secret "maas-api-e2e-api-b-e7ca74-serving-cert" not found Nov 03 13:32:40.177539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.177517 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddd6f5e-23e4-4778-909b-79a2ff540375-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.179302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.179271 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-metrics-certs\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.263143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.261441 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2xn\" (UniqueName: \"kubernetes.io/projected/2ddd6f5e-23e4-4778-909b-79a2ff540375-kube-api-access-4z2xn\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.281721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.281683 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp"] Nov 03 13:32:40.295207 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.295171 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8"] Nov 03 13:32:40.295410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.295367 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.299767 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.299747 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.305250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.305224 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp"] Nov 03 13:32:40.356095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.355195 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:40.361126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.359553 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8"] Nov 03 13:32:40.381301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381265 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2135eb-5813-4588-ac68-317b47fca015-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.381447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381312 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.381447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381410 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.381538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381460 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.381576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381530 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.381576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381567 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.381689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381670 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2135eb-5813-4588-ac68-317b47fca015-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.381752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381714 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.381800 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381748 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.381800 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381785 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.381905 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381879 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.381982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.381966 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.382041 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.382011 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.382181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.382161 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.382222 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.382193 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.382272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.382227 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvz5\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-kube-api-access-bcvz5\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.382337 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.382294 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjm7x\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-kube-api-access-vjm7x\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.382387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.382349 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.484014 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.483963 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2135eb-5813-4588-ac68-317b47fca015-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.484227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484023 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.484227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484058 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.484227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484104 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.484227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484147 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484289 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484347 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484453 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484476 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484511 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvz5\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-kube-api-access-bcvz5\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484542 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjm7x\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-kube-api-access-vjm7x\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484547 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484585 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484600 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484599 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484695 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2135eb-5813-4588-ac68-317b47fca015-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484736 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484766 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484791 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484812 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.485387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.484841 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.486248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.485537 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.486248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.485759 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.486248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.485808 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.486248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.485816 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.486478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.486289 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2135eb-5813-4588-ac68-317b47fca015-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.486478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.486447 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.486478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.486456 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.487503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.487476 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.487621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.487557 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2135eb-5813-4588-ac68-317b47fca015-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.488022 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.487996 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.488834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.488813 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.535622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.535554 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:40.539533 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.539507 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.539954 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.539936 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjm7x\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-kube-api-access-vjm7x\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.540000 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.539950 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.558084 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.558044 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvz5\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-kube-api-access-bcvz5\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.615822 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.615791 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:40.621331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.621307 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:40.691135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.687910 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.692641 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.692597 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls\") pod \"maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:40.908211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:40.908178 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:41.044635 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:32:41.044593 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2cca85_c79f_4f8b_9852_06ea26c6f2fa.slice/crio-1acf80058e9d0e01d9474a159392c9b7e9d7154b0abb025b00a36bef2a49ec44 WatchSource:0}: Error finding container 1acf80058e9d0e01d9474a159392c9b7e9d7154b0abb025b00a36bef2a49ec44: Status 404 returned error can't find the container with id 1acf80058e9d0e01d9474a159392c9b7e9d7154b0abb025b00a36bef2a49ec44 Nov 03 13:32:41.045301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.045273 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp"] Nov 03 13:32:41.047365 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.047329 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.047473 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.047405 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.047473 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.047450 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.050854 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.050830 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8"] Nov 03 13:32:41.080125 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.080052 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.080241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.080156 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.080241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.080192 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.293697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.288361 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" event={"ID":"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa","Type":"ContainerStarted","Data":"4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb"} Nov 03 13:32:41.293697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.288436 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" event={"ID":"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa","Type":"ContainerStarted","Data":"1acf80058e9d0e01d9474a159392c9b7e9d7154b0abb025b00a36bef2a49ec44"} Nov 03 13:32:41.293697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.290206 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" event={"ID":"5b2135eb-5813-4588-ac68-317b47fca015","Type":"ContainerStarted","Data":"07f1642436acdd8855e4252564d5f6d88fc767a4d41f8a44076fbb8a8f08cb75"} Nov 03 13:32:41.293697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.290241 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" event={"ID":"5b2135eb-5813-4588-ac68-317b47fca015","Type":"ContainerStarted","Data":"0dc51deea33f2d462ef34197ea18beb2eceed2cadd204111997ba11480c5d4a5"} Nov 03 13:32:41.303455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.303428 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn"] Nov 03 13:32:41.304549 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:32:41.304524 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddd6f5e_23e4_4778_909b_79a2ff540375.slice/crio-50a81f1e3c2ca4946b8640add483a31b665ce9b11881e838586ca693c4c1857f WatchSource:0}: Error finding container 50a81f1e3c2ca4946b8640add483a31b665ce9b11881e838586ca693c4c1857f: Status 404 returned error can't find the container with id 50a81f1e3c2ca4946b8640add483a31b665ce9b11881e838586ca693c4c1857f Nov 03 13:32:41.407211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.407159 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" podStartSLOduration=1.407141221 podStartE2EDuration="1.407141221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:41.404105248 +0000 UTC m=+1694.598508835" watchObservedRunningTime="2026-11-03 13:32:41.407141221 +0000 UTC m=+1694.601544814" Nov 03 13:32:41.557798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.557678 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" podStartSLOduration=1.557659184 podStartE2EDuration="1.557659184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:41.55300375 +0000 UTC m=+1694.747407355" watchObservedRunningTime="2026-11-03 13:32:41.557659184 +0000 UTC m=+1694.752062780" Nov 03 13:32:41.621740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.621702 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:41.621938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:41.621768 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:42.180095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:42.179604 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:42.318235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:42.318192 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" event={"ID":"2ddd6f5e-23e4-4778-909b-79a2ff540375","Type":"ContainerStarted","Data":"50a81f1e3c2ca4946b8640add483a31b665ce9b11881e838586ca693c4c1857f"} Nov 03 13:32:42.624169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:42.624124 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:32:42.624717 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:42.624383 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 13:32:42.624988 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:42.624959 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:32:42.625185 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:42.625154 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 13:32:43.376308 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:43.376251 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" event={"ID":"2ddd6f5e-23e4-4778-909b-79a2ff540375","Type":"ContainerStarted","Data":"6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24"} Nov 03 13:32:43.389121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:43.383086 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:43.501339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:43.500667 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" podStartSLOduration=3.28702718 podStartE2EDuration="4.500646431s" totalImagesPullingTime="1.213619251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:43.499479076 +0000 UTC m=+1696.693882671" watchObservedRunningTime="2026-11-03 13:32:43.500646431 +0000 UTC m=+1696.695050027" Nov 03 13:32:43.617575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:43.616874 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:32:43.617575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:43.616937 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 13:32:43.623627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:43.622878 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:32:43.623627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:43.622945 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 13:32:44.616788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:44.616749 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:32:44.617192 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:44.616809 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 13:32:44.622137 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:44.622112 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:32:44.622246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:44.622159 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 13:32:45.612792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:45.612740 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:45.617225 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:45.616946 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:32:45.617225 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:45.617014 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 13:32:45.622357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:45.622329 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:32:45.622653 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:45.622381 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 13:32:45.775243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:45.775203 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:45.928098 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:45.924664 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:46.616398 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:46.616365 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:32:46.616575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:46.616416 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 13:32:46.621618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:46.621592 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:32:46.621925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:46.621641 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 13:32:47.622276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:47.621785 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:32:47.622276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:47.621845 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 13:32:47.623026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:47.623004 4924 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:32:47.623184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:47.623161 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 13:32:48.578685 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:32:48.578610 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:32:48.630556 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:48.629157 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:48.630556 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:48.630490 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:48.631965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:48.631943 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:32:48.642093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:48.638349 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:48.642093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:48.639471 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:48.642332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:48.642299 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:32:50.426407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:50.426379 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:32:50.994775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:50.994740 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:32:56.091829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:32:56.091780 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:01.264890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.264857 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69"] Nov 03 13:33:01.268793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.268772 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.334787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.334733 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.334985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.334805 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxn7\" (UniqueName: \"kubernetes.io/projected/df36687a-57bd-4b91-8b5f-e8f095f700a9-kube-api-access-drxn7\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.334985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.334932 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36687a-57bd-4b91-8b5f-e8f095f700a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.335142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.335099 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.366603 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.366552 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-be8d0a-serving-cert\"" Nov 03 13:33:01.372862 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.372825 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69"] Nov 03 13:33:01.437169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.437130 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.437362 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.437229 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.437362 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.437268 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxn7\" (UniqueName: \"kubernetes.io/projected/df36687a-57bd-4b91-8b5f-e8f095f700a9-kube-api-access-drxn7\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.437362 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:33:01.437302 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-serving-cert: secret "maas-api-e2e-ipp-a-be8d0a-serving-cert" not found Nov 03 13:33:01.437525 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:33:01.437384 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls podName:df36687a-57bd-4b91-8b5f-e8f095f700a9 nodeName:}" failed. No retries permitted until 2026-11-03 13:33:01.937363736 +0000 UTC m=+1715.131767325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls") pod "maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" (UID: "df36687a-57bd-4b91-8b5f-e8f095f700a9") : secret "maas-api-e2e-ipp-a-be8d0a-serving-cert" not found Nov 03 13:33:01.437525 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.437429 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36687a-57bd-4b91-8b5f-e8f095f700a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.438340 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.438309 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36687a-57bd-4b91-8b5f-e8f095f700a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.440472 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.440452 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.468989 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.468959 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:01.492656 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.492624 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxn7\" (UniqueName: \"kubernetes.io/projected/df36687a-57bd-4b91-8b5f-e8f095f700a9-kube-api-access-drxn7\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.701427 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.701391 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:01.847266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.847229 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:01.949805 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.949764 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:01.952740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:01.952674 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:02.183013 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:02.182976 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:02.558334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:02.558302 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69"] Nov 03 13:33:02.561403 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:33:02.561362 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf36687a_57bd_4b91_8b5f_e8f095f700a9.slice/crio-aaabba0d0232336572ef22e9d7e59950e47a15ceb8d30bf34d0e8b338df986c5 WatchSource:0}: Error finding container aaabba0d0232336572ef22e9d7e59950e47a15ceb8d30bf34d0e8b338df986c5: Status 404 returned error can't find the container with id aaabba0d0232336572ef22e9d7e59950e47a15ceb8d30bf34d0e8b338df986c5 Nov 03 13:33:02.590901 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:02.590863 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" event={"ID":"df36687a-57bd-4b91-8b5f-e8f095f700a9","Type":"ContainerStarted","Data":"aaabba0d0232336572ef22e9d7e59950e47a15ceb8d30bf34d0e8b338df986c5"} Nov 03 13:33:04.646093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:04.643500 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" event={"ID":"df36687a-57bd-4b91-8b5f-e8f095f700a9","Type":"ContainerStarted","Data":"21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046"} Nov 03 13:33:04.646093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:04.644547 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:07.248508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:07.248473 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:11.720758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:11.720728 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:33:11.835619 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:11.835563 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" podStartSLOduration=9.853846208 podStartE2EDuration="10.835545713s" totalImagesPullingTime="981.699505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:04.752178276 +0000 UTC m=+1717.946581872" watchObservedRunningTime="2026-11-03 13:33:11.835545713 +0000 UTC m=+1725.029949307" Nov 03 13:33:12.666371 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:12.666339 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:16.033113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:16.033081 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:17.857761 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:17.857723 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:18.354827 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:18.354786 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:18.528951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:18.528916 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:18.559392 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:33:18.559345 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:33:20.999410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:20.999373 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv"] Nov 03 13:33:21.094449 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:21.094405 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:23.341607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:23.341570 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:26.355826 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:26.355768 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:28.538322 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:28.538273 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:33.697727 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:33.697689 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:36.020987 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.020938 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" podUID="9542ec83-268f-48d4-a677-4b42c3ac9c5f" containerName="main" containerID="cri-o://208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809" gracePeriod=45 Nov 03 13:33:36.310908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.310877 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:33:36.473256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.472164 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tmp-dir\") pod \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " Nov 03 13:33:36.473256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.472237 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-model-cache\") pod \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " Nov 03 13:33:36.473256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.472308 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tls-certs\") pod \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " Nov 03 13:33:36.473256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.472385 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-home\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-home\") pod \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " Nov 03 13:33:36.473256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.472580 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-dshm\") pod \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " Nov 03 13:33:36.473256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.472645 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9542ec83-268f-48d4-a677-4b42c3ac9c5f-kube-api-access-xqtgb\" (UniqueName: \"kubernetes.io/projected/9542ec83-268f-48d4-a677-4b42c3ac9c5f-kube-api-access-xqtgb\") pod \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\" (UID: \"9542ec83-268f-48d4-a677-4b42c3ac9c5f\") " Nov 03 13:33:36.475885 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.474776 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-home" pod "9542ec83-268f-48d4-a677-4b42c3ac9c5f" (UID: "9542ec83-268f-48d4-a677-4b42c3ac9c5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:36.475885 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.480152 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-model-cache" pod "9542ec83-268f-48d4-a677-4b42c3ac9c5f" (UID: "9542ec83-268f-48d4-a677-4b42c3ac9c5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:36.475885 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.480215 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tmp-dir" pod "9542ec83-268f-48d4-a677-4b42c3ac9c5f" (UID: "9542ec83-268f-48d4-a677-4b42c3ac9c5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:36.483316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.483283 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-dshm" pod "9542ec83-268f-48d4-a677-4b42c3ac9c5f" (UID: "9542ec83-268f-48d4-a677-4b42c3ac9c5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:36.489992 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.489965 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tls-certs" pod "9542ec83-268f-48d4-a677-4b42c3ac9c5f" (UID: "9542ec83-268f-48d4-a677-4b42c3ac9c5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:33:36.490287 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.490233 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9542ec83-268f-48d4-a677-4b42c3ac9c5f-kube-api-access-xqtgb" pod "9542ec83-268f-48d4-a677-4b42c3ac9c5f" (UID: "9542ec83-268f-48d4-a677-4b42c3ac9c5f"). InnerVolumeSpecName "kube-api-access-xqtgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:36.575060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.574968 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:36.575060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.574996 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:36.575060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.575006 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqtgb\" (UniqueName: \"kubernetes.io/projected/9542ec83-268f-48d4-a677-4b42c3ac9c5f-kube-api-access-xqtgb\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:36.575060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.575015 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:36.575060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.575039 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9542ec83-268f-48d4-a677-4b42c3ac9c5f-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:36.575060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.575048 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9542ec83-268f-48d4-a677-4b42c3ac9c5f-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:36.941165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.941125 4924 generic.go:356] "Generic (PLEG): container finished" podID="9542ec83-268f-48d4-a677-4b42c3ac9c5f" containerID="208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809" exitCode=0 Nov 03 13:33:36.941361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.941181 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" Nov 03 13:33:36.941361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.941208 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" event={"ID":"9542ec83-268f-48d4-a677-4b42c3ac9c5f","Type":"ContainerDied","Data":"208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809"} Nov 03 13:33:36.941361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.941254 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv" event={"ID":"9542ec83-268f-48d4-a677-4b42c3ac9c5f","Type":"ContainerDied","Data":"b57dff49c014a8e25aa4d7d7abe4d36da3d805ec5515f149555769ea6232e06c"} Nov 03 13:33:36.941361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.941275 4924 scope.go:122] "RemoveContainer" containerID="208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809" Nov 03 13:33:36.953257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.953237 4924 scope.go:122] "RemoveContainer" containerID="208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809" Nov 03 13:33:36.953576 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:33:36.953551 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809\": container with ID starting with 208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809 not found: ID does not exist" containerID="208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809" Nov 03 13:33:36.953661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:36.953589 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809"} err="failed to get container status \"208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809\": rpc error: code = NotFound desc = could not find container \"208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809\": container with ID starting with 208cac9a73d85403d0f1b45916f9d8a3dcc0c5ec2f6ddb5eb545fa0ac508c809 not found: ID does not exist" Nov 03 13:33:37.102129 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:37.102095 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv"] Nov 03 13:33:37.148720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:37.148683 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5295ab29/e2e-lifecycle-model-5295ab29-kserve-7bf8d7dbf-rd7zv"] Nov 03 13:33:38.284342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:38.284307 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:38.952413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:38.952381 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9542ec83-268f-48d4-a677-4b42c3ac9c5f" path="/var/lib/kubelet/pods/9542ec83-268f-48d4-a677-4b42c3ac9c5f/volumes" Nov 03 13:33:40.273678 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:40.273613 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:41.048254 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:41.048190 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:41.183478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:41.183437 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:42.696365 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:42.696329 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:43.270725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:43.270628 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:43.273658 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:43.273619 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c"] Nov 03 13:33:43.273952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:43.273912 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" containerID="cri-o://0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8" gracePeriod=30 Nov 03 13:33:43.290223 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:43.290192 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v"] Nov 03 13:33:43.290406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:43.290385 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" containerID="cri-o://aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40" gracePeriod=30 Nov 03 13:33:44.498257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:44.498221 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:45.927000 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:45.926964 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:46.057820 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:46.057789 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:46.190763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:46.190675 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:48.604815 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.604784 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:33:48.604815 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:33:48.604801 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:33:48.638699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.638676 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:33:48.686105 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686040 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-data\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.686105 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686109 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-istio-token\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-istio-token\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.686334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686161 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-envoy\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.686334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686206 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-socket\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.686334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686268 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-data" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.686334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686281 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956c58e2-4740-418d-8bdc-16ed0155d895-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c58e2-4740-418d-8bdc-16ed0155d895-istiod-ca-cert\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.686548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686344 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-podinfo\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.686548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686394 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-token\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-token\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.686548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686474 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istiod-ca-cert\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.686548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686477 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-socket" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.686747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686558 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-certs\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.686747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686641 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-envoy\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.686747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686668 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956c58e2-4740-418d-8bdc-16ed0155d895-istiod-ca-cert" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:48.686747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686688 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-kube-api-access-hp5bz\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-kube-api-access-hp5bz\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.686747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686740 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-credential-socket\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.687054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686776 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-data\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.687054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686829 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-credential-socket\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.687054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686856 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istiod-ca-cert" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:48.687054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686868 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/956c58e2-4740-418d-8bdc-16ed0155d895-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c58e2-4740-418d-8bdc-16ed0155d895-istio-podinfo\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.687054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686906 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-socket\") pod \"956c58e2-4740-418d-8bdc-16ed0155d895\" (UID: \"956c58e2-4740-418d-8bdc-16ed0155d895\") " Nov 03 13:33:48.687054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686950 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-certs\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.687054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686961 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-certs" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.687054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.686980 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-kube-api-access-bn2mk\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-kube-api-access-bn2mk\") pod \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\" (UID: \"e8067e6f-06cc-48c5-8e3e-af27a06a8f87\") " Nov 03 13:33:48.687443 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687136 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-credential-socket" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.687496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687439 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-data" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687608 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687627 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-certs" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687633 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c58e2-4740-418d-8bdc-16ed0155d895-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687669 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687688 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687703 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687717 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687731 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687808 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-socket" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.687845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.687834 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-credential-socket" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.688853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.688826 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-envoy" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.688962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.688879 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-istio-token" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.689587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.689552 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/956c58e2-4740-418d-8bdc-16ed0155d895-istio-podinfo" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:48.689864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.689795 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-kube-api-access-hp5bz" pod "956c58e2-4740-418d-8bdc-16ed0155d895" (UID: "956c58e2-4740-418d-8bdc-16ed0155d895"). InnerVolumeSpecName "kube-api-access-hp5bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.689924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.689906 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-podinfo" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:48.689961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.689909 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-kube-api-access-bn2mk" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "kube-api-access-bn2mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.689995 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.689984 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-envoy" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.690563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.690547 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-token" pod "e8067e6f-06cc-48c5-8e3e-af27a06a8f87" (UID: "e8067e6f-06cc-48c5-8e3e-af27a06a8f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.788957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.788916 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5bz\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-kube-api-access-hp5bz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.788957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.788957 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.788975 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c58e2-4740-418d-8bdc-16ed0155d895-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.788989 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.789004 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.789016 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2mk\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-kube-api-access-bn2mk\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.789031 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c58e2-4740-418d-8bdc-16ed0155d895-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.789044 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c58e2-4740-418d-8bdc-16ed0155d895-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.789057 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.789085 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.789230 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:48.789099 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8067e6f-06cc-48c5-8e3e-af27a06a8f87-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:49.031873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.031835 4924 generic.go:356] "Generic (PLEG): container finished" podID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerID="aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40" exitCode=0 Nov 03 13:33:49.032102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.031897 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" Nov 03 13:33:49.032102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.031922 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" event={"ID":"e8067e6f-06cc-48c5-8e3e-af27a06a8f87","Type":"ContainerDied","Data":"aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40"} Nov 03 13:33:49.032102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.031991 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v" event={"ID":"e8067e6f-06cc-48c5-8e3e-af27a06a8f87","Type":"ContainerDied","Data":"17b68e0aa110882792ea092649b87d26c69a8fd1c93a159b59e6df691fd7f794"} Nov 03 13:33:49.032102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.032025 4924 scope.go:122] "RemoveContainer" containerID="aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40" Nov 03 13:33:49.033519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.033496 4924 generic.go:356] "Generic (PLEG): container finished" podID="956c58e2-4740-418d-8bdc-16ed0155d895" containerID="0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8" exitCode=0 Nov 03 13:33:49.033635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.033609 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" Nov 03 13:33:49.033692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.033654 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" event={"ID":"956c58e2-4740-418d-8bdc-16ed0155d895","Type":"ContainerDied","Data":"0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8"} Nov 03 13:33:49.033755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.033705 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c" event={"ID":"956c58e2-4740-418d-8bdc-16ed0155d895","Type":"ContainerDied","Data":"48c00c0b59818b1ac7e696ab55072d973d765699744fb1d6794d213fe2a5a32b"} Nov 03 13:33:49.042821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.042799 4924 scope.go:122] "RemoveContainer" containerID="aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40" Nov 03 13:33:49.043162 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:33:49.043140 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40\": container with ID starting with aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40 not found: ID does not exist" containerID="aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40" Nov 03 13:33:49.043250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.043170 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40"} err="failed to get container status \"aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40\": rpc error: code = NotFound desc = could not find container \"aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40\": container with ID starting with aea3bc9a05c14f10ab0c63b05e61dcc7ba3a89ea1efe9918b535ea5cfcac8c40 not found: ID does not exist" Nov 03 13:33:49.043250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.043187 4924 scope.go:122] "RemoveContainer" containerID="0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8" Nov 03 13:33:49.052966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.052942 4924 scope.go:122] "RemoveContainer" containerID="0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8" Nov 03 13:33:49.053317 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:33:49.053293 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8\": container with ID starting with 0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8 not found: ID does not exist" containerID="0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8" Nov 03 13:33:49.053387 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.053327 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8"} err="failed to get container status \"0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8\": rpc error: code = NotFound desc = could not find container \"0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8\": container with ID starting with 0257a74f7fe05e81e80ac3fa9cbf41a26cc83471bb4cf8298bab1d363e20e8f8 not found: ID does not exist" Nov 03 13:33:49.188646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.188587 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c"] Nov 03 13:33:49.193339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.193310 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-7426c"] Nov 03 13:33:49.296817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.296295 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v"] Nov 03 13:33:49.300497 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:49.300467 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-2xs2v"] Nov 03 13:33:50.094509 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:50.094475 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:50.284279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:50.284240 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:50.414260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:50.414227 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:50.864195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:50.864107 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:50.955567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:50.955529 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" path="/var/lib/kubelet/pods/956c58e2-4740-418d-8bdc-16ed0155d895/volumes" Nov 03 13:33:50.956662 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:50.956632 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" path="/var/lib/kubelet/pods/e8067e6f-06cc-48c5-8e3e-af27a06a8f87/volumes" Nov 03 13:33:51.549844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:51.549791 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:52.351591 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:52.351535 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:52.957147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:52.957111 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:53.863915 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:53.863884 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:54.762250 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:54.762217 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:54.923370 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:54.923277 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:55.438888 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:55.438841 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:55.569002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:55.568961 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:56.270681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:56.270636 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:56.414998 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:56.414960 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:56.557032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:56.556912 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:57.108882 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:57.108850 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:57.809686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:57.809639 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:57.940152 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:57.940117 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:58.610571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:58.610529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:59.395096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:59.395017 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:59.611325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:59.611292 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:33:59.774797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:33:59.774762 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:00.434226 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:00.434183 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:00.631464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:00.631430 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:02.910447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:02.910409 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:03.262699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:03.262613 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:05.214517 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:05.214441 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:06.636935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:06.636897 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:08.793980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:08.793946 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:08.933620 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:08.933583 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:09.173610 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:09.173572 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:11.712004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:11.711964 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:12.561209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:12.561168 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:13.296200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:13.284223 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:13.452748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:13.442981 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:14.595481 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:14.595445 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:15.274016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:15.273983 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:15.410750 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:15.410720 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:18.497523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:18.497476 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:18.561507 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:34:18.561469 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:34:18.790321 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:18.790243 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:18.958374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:18.958341 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:20.222257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:20.222221 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:21.804013 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:21.803977 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:21.985488 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:21.985451 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:25.841220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:25.841186 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:26.696662 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:26.696626 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:27.221254 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:27.221226 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:34:27.225003 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:27.224977 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:34:30.568523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:30.568485 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h"] Nov 03 13:34:30.568980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:30.568686 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" podUID="0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" containerName="maas-api" containerID="cri-o://90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96" gracePeriod=30 Nov 03 13:34:30.669181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:30.669100 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:30.968113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:30.965023 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:34:31.032097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.031531 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-metrics-certs\") pod \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " Nov 03 13:34:31.032097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.031733 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls\") pod \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " Nov 03 13:34:31.032097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.031873 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-openshift-service-ca\") pod \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " Nov 03 13:34:31.032097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.031908 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-kube-api-access-hpvmz\" (UniqueName: \"kubernetes.io/projected/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-kube-api-access-hpvmz\") pod \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\" (UID: \"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7\") " Nov 03 13:34:31.041249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.035717 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-openshift-service-ca" pod "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" (UID: "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:31.061654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.052251 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-metrics-certs" pod "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" (UID: "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:34:31.061654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.052485 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-kube-api-access-hpvmz" pod "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" (UID: "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7"). InnerVolumeSpecName "kube-api-access-hpvmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:31.062455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.062423 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls" pod "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" (UID: "0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:34:31.134231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.134124 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.134231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.134173 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.134231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.134190 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpvmz\" (UniqueName: \"kubernetes.io/projected/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-kube-api-access-hpvmz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.134231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.134205 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.705095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.701518 4924 generic.go:356] "Generic (PLEG): container finished" podID="0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" containerID="90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96" exitCode=0 Nov 03 13:34:31.705095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.701768 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" event={"ID":"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7","Type":"ContainerDied","Data":"90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96"} Nov 03 13:34:31.705095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.701821 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" event={"ID":"0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7","Type":"ContainerDied","Data":"07fa679344c862c2eedda150a7ea3c750087e3cf4b0c791090e4ebe380d4e59a"} Nov 03 13:34:31.705095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.701852 4924 scope.go:122] "RemoveContainer" containerID="90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96" Nov 03 13:34:31.705095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.702097 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h" Nov 03 13:34:31.738556 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.738524 4924 scope.go:122] "RemoveContainer" containerID="90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96" Nov 03 13:34:31.738949 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:34:31.738921 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96\": container with ID starting with 90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96 not found: ID does not exist" containerID="90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96" Nov 03 13:34:31.739016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.738964 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96"} err="failed to get container status \"90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96\": rpc error: code = NotFound desc = could not find container \"90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96\": container with ID starting with 90c747c42a50e3b1d27a6c257a093cbac922490d67b5643a16a1c133abec4d96 not found: ID does not exist" Nov 03 13:34:31.860710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.860612 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h"] Nov 03 13:34:31.906543 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:31.906504 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-461b2d-6b64f6c89f-x5z7h"] Nov 03 13:34:32.683121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:32.683061 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:32.953181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:32.953096 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" path="/var/lib/kubelet/pods/0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7/volumes" Nov 03 13:34:38.267563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:38.267526 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:41.174548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:41.174502 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:41.314195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:41.314160 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:42.809305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:42.809269 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:43.369513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:43.369471 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:43.891502 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:43.891468 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:44.193309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:44.193277 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:47.951513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:47.951474 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:48.564593 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:34:48.564541 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:34:48.599594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:48.599557 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:51.409249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:51.409208 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:51.775099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:51.774997 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:51.946048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:51.946007 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:52.084309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:52.084206 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:54.261868 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:54.261831 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:54.410546 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:54.410514 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:57.039271 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:57.039230 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:34:57.695775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:34:57.695720 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:02.275661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.275623 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s"] Nov 03 13:35:02.276219 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.275857 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" podUID="a2935bad-4728-4a81-8a5a-1bb52e55099d" containerName="maas-api" containerID="cri-o://004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a" gracePeriod=30 Nov 03 13:35:02.363866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.363825 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:02.772091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.768268 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:35:02.905890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.905801 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-maas-api-tls\") pod \"a2935bad-4728-4a81-8a5a-1bb52e55099d\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " Nov 03 13:35:02.906066 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.905887 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2935bad-4728-4a81-8a5a-1bb52e55099d-kube-api-access-dtls2\" (UniqueName: \"kubernetes.io/projected/a2935bad-4728-4a81-8a5a-1bb52e55099d-kube-api-access-dtls2\") pod \"a2935bad-4728-4a81-8a5a-1bb52e55099d\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " Nov 03 13:35:02.906066 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.906066 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2935bad-4728-4a81-8a5a-1bb52e55099d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2935bad-4728-4a81-8a5a-1bb52e55099d-openshift-service-ca\") pod \"a2935bad-4728-4a81-8a5a-1bb52e55099d\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " Nov 03 13:35:02.906274 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.906121 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-metrics-certs\") pod \"a2935bad-4728-4a81-8a5a-1bb52e55099d\" (UID: \"a2935bad-4728-4a81-8a5a-1bb52e55099d\") " Nov 03 13:35:02.906494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.906459 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2935bad-4728-4a81-8a5a-1bb52e55099d-openshift-service-ca" pod "a2935bad-4728-4a81-8a5a-1bb52e55099d" (UID: "a2935bad-4728-4a81-8a5a-1bb52e55099d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:02.906686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.906663 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2935bad-4728-4a81-8a5a-1bb52e55099d-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:02.908449 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.908416 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-metrics-certs" pod "a2935bad-4728-4a81-8a5a-1bb52e55099d" (UID: "a2935bad-4728-4a81-8a5a-1bb52e55099d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:35:02.908880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.908844 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2935bad-4728-4a81-8a5a-1bb52e55099d-kube-api-access-dtls2" pod "a2935bad-4728-4a81-8a5a-1bb52e55099d" (UID: "a2935bad-4728-4a81-8a5a-1bb52e55099d"). InnerVolumeSpecName "kube-api-access-dtls2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:02.909127 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.909103 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-maas-api-tls" pod "a2935bad-4728-4a81-8a5a-1bb52e55099d" (UID: "a2935bad-4728-4a81-8a5a-1bb52e55099d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:35:02.953939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.953899 4924 generic.go:356] "Generic (PLEG): container finished" podID="a2935bad-4728-4a81-8a5a-1bb52e55099d" containerID="004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a" exitCode=0 Nov 03 13:35:02.954142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.953940 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" Nov 03 13:35:02.954142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.953948 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" event={"ID":"a2935bad-4728-4a81-8a5a-1bb52e55099d","Type":"ContainerDied","Data":"004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a"} Nov 03 13:35:02.954142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.953992 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s" event={"ID":"a2935bad-4728-4a81-8a5a-1bb52e55099d","Type":"ContainerDied","Data":"11366c4dc99c06fbc39d1e05ced0594cc547b92bb6a2bbd66bff3e36a66ab027"} Nov 03 13:35:02.954142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.954018 4924 scope.go:122] "RemoveContainer" containerID="004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a" Nov 03 13:35:02.966153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.966124 4924 scope.go:122] "RemoveContainer" containerID="004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a" Nov 03 13:35:02.966464 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:35:02.966447 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a\": container with ID starting with 004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a not found: ID does not exist" containerID="004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a" Nov 03 13:35:02.966547 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:02.966481 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a"} err="failed to get container status \"004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a\": rpc error: code = NotFound desc = could not find container \"004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a\": container with ID starting with 004f795ae03dc0a168084ebbc56f9d82649ea9662030cdd810db5ea101b5454a not found: ID does not exist" Nov 03 13:35:03.008839 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:03.008802 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:03.009063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:03.009037 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2935bad-4728-4a81-8a5a-1bb52e55099d-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:03.009063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:03.009064 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtls2\" (UniqueName: \"kubernetes.io/projected/a2935bad-4728-4a81-8a5a-1bb52e55099d-kube-api-access-dtls2\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:03.112118 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:03.112055 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s"] Nov 03 13:35:03.120413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:03.120381 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5295ab29-5d8cf4ff8b-nd47s"] Nov 03 13:35:04.085166 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.085134 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:04.242785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.242748 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:04.369052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.368966 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:04.667100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.667050 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp"] Nov 03 13:35:04.667354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.667328 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" containerID="cri-o://4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb" gracePeriod=30 Nov 03 13:35:04.673122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.673056 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:04.697297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.697254 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8"] Nov 03 13:35:04.697759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.697722 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" containerID="cri-o://07f1642436acdd8855e4252564d5f6d88fc767a4d41f8a44076fbb8a8f08cb75" gracePeriod=30 Nov 03 13:35:04.700826 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.700802 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7"] Nov 03 13:35:04.700990 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.700972 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" containerID="cri-o://90b97bff2bbca93cb53c5c1b49a52ba5ac887d3986cd407b9d23de99d8379888" gracePeriod=30 Nov 03 13:35:04.954720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:04.954625 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2935bad-4728-4a81-8a5a-1bb52e55099d" path="/var/lib/kubelet/pods/a2935bad-4728-4a81-8a5a-1bb52e55099d/volumes" Nov 03 13:35:06.365036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:06.364963 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:06.576813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:06.576779 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:09.536951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:09.536860 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2"] Nov 03 13:35:09.537620 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:09.537583 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" containerID="cri-o://40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff" gracePeriod=30 Nov 03 13:35:10.005989 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.005157 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:35:10.010752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.010722 4924 generic.go:356] "Generic (PLEG): container finished" podID="7dff8111-1807-47fd-a147-af28bae5cf12" containerID="90b97bff2bbca93cb53c5c1b49a52ba5ac887d3986cd407b9d23de99d8379888" exitCode=0 Nov 03 13:35:10.010897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.010798 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" event={"ID":"7dff8111-1807-47fd-a147-af28bae5cf12","Type":"ContainerDied","Data":"90b97bff2bbca93cb53c5c1b49a52ba5ac887d3986cd407b9d23de99d8379888"} Nov 03 13:35:10.010897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.010874 4924 scope.go:122] "RemoveContainer" containerID="2625bfbe7ec29b82a2841f10c40e68ede4fbaca19168abff1a364b4edfe82e6a" Nov 03 13:35:10.013294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.013267 4924 generic.go:356] "Generic (PLEG): container finished" podID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerID="4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb" exitCode=0 Nov 03 13:35:10.013412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.013322 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" Nov 03 13:35:10.013412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.013346 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" event={"ID":"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa","Type":"ContainerDied","Data":"4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb"} Nov 03 13:35:10.013412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.013397 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp" event={"ID":"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa","Type":"ContainerDied","Data":"1acf80058e9d0e01d9474a159392c9b7e9d7154b0abb025b00a36bef2a49ec44"} Nov 03 13:35:10.015243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.015223 4924 generic.go:356] "Generic (PLEG): container finished" podID="5b2135eb-5813-4588-ac68-317b47fca015" containerID="07f1642436acdd8855e4252564d5f6d88fc767a4d41f8a44076fbb8a8f08cb75" exitCode=0 Nov 03 13:35:10.015370 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.015354 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" event={"ID":"5b2135eb-5813-4588-ac68-317b47fca015","Type":"ContainerDied","Data":"07f1642436acdd8855e4252564d5f6d88fc767a4d41f8a44076fbb8a8f08cb75"} Nov 03 13:35:10.026295 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.026270 4924 scope.go:122] "RemoveContainer" containerID="4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb" Nov 03 13:35:10.049172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.049152 4924 scope.go:122] "RemoveContainer" containerID="4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb" Nov 03 13:35:10.049446 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:35:10.049427 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb\": container with ID starting with 4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb not found: ID does not exist" containerID="4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb" Nov 03 13:35:10.049516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.049456 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb"} err="failed to get container status \"4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb\": rpc error: code = NotFound desc = could not find container \"4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb\": container with ID starting with 4a80878ef6e0a16d2cb04ef62af8627b6a601b8f5516e50e02e1df5bed150ddb not found: ID does not exist" Nov 03 13:35:10.052730 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.052708 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:35:10.054259 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054238 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-kube-api-access-bcvz5\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-kube-api-access-bcvz5\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054274 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-credential-socket\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054325 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-certs\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054404 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-socket\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054477 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054433 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-data\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054476 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-envoy\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054506 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istiod-ca-cert\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054629 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-credential-socket" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.054650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054642 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-podinfo\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054677 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-token\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-token\") pod \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\" (UID: \"7e2cca85-c79f-4f8b-9852-06ea26c6f2fa\") " Nov 03 13:35:10.054758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054712 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-certs" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.054758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054729 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-socket" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.054758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054744 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-data" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.055001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.054980 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istiod-ca-cert" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.055212 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.055192 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:35:10.055323 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.055299 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.055388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.055322 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.055388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.055337 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.055388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.055351 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.055388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.055364 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.056899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.056837 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-envoy" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.056899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.056847 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-podinfo" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.057049 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.056901 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-token" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.057164 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.057146 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-kube-api-access-bcvz5" pod "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" (UID: "7e2cca85-c79f-4f8b-9852-06ea26c6f2fa"). InnerVolumeSpecName "kube-api-access-bcvz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.155748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.155706 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-istio-token\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-istio-token\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.155936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.155782 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-envoy\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.155936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.155838 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-data\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.155936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.155889 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-data\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.156124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.155943 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-kube-api-access-8smdw\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-kube-api-access-8smdw\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156187 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-data" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156229 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-data" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156297 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-socket\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156349 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-istio-token\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-istio-token\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156394 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b2135eb-5813-4588-ac68-317b47fca015-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2135eb-5813-4588-ac68-317b47fca015-istio-podinfo\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156427 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-credential-socket\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156458 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-envoy\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156738 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-socket" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156754 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-credential-socket" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.156521 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-socket\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.157134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157119 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-socket" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.157804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157205 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dff8111-1807-47fd-a147-af28bae5cf12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff8111-1807-47fd-a147-af28bae5cf12-istio-podinfo\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.157804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157538 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-certs\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.157804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157621 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-credential-socket\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.157804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157652 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-certs\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.157804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157705 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b2135eb-5813-4588-ac68-317b47fca015-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2135eb-5813-4588-ac68-317b47fca015-istiod-ca-cert\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.157804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157769 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-kube-api-access-vjm7x\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-kube-api-access-vjm7x\") pod \"5b2135eb-5813-4588-ac68-317b47fca015\" (UID: \"5b2135eb-5813-4588-ac68-317b47fca015\") " Nov 03 13:35:10.158141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157804 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-certs" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.158141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.157816 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dff8111-1807-47fd-a147-af28bae5cf12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff8111-1807-47fd-a147-af28bae5cf12-istiod-ca-cert\") pod \"7dff8111-1807-47fd-a147-af28bae5cf12\" (UID: \"7dff8111-1807-47fd-a147-af28bae5cf12\") " Nov 03 13:35:10.158141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158017 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-certs" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.158411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158385 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dff8111-1807-47fd-a147-af28bae5cf12-istiod-ca-cert" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.158411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158381 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b2135eb-5813-4588-ac68-317b47fca015-istiod-ca-cert" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.158605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158568 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-istio-token" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.158690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158670 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-credential-socket" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.158880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158862 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158883 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcvz5\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-kube-api-access-bcvz5\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158899 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158912 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158925 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158931 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-envoy" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158939 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158955 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158967 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158975 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158984 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.158996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.158994 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.159622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.159006 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.159622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.159015 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b2135eb-5813-4588-ac68-317b47fca015-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.159622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.159024 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff8111-1807-47fd-a147-af28bae5cf12-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.159622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.159036 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.160320 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.160284 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dff8111-1807-47fd-a147-af28bae5cf12-istio-podinfo" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.160482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.160457 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b2135eb-5813-4588-ac68-317b47fca015-istio-podinfo" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.160680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.160656 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-envoy" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.160780 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.160672 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-istio-token" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.161332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.161312 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-kube-api-access-8smdw" pod "7dff8111-1807-47fd-a147-af28bae5cf12" (UID: "7dff8111-1807-47fd-a147-af28bae5cf12"). InnerVolumeSpecName "kube-api-access-8smdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.162331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.162298 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-kube-api-access-vjm7x" pod "5b2135eb-5813-4588-ac68-317b47fca015" (UID: "5b2135eb-5813-4588-ac68-317b47fca015"). InnerVolumeSpecName "kube-api-access-vjm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.260612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.260457 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b2135eb-5813-4588-ac68-317b47fca015-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.260612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.260496 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff8111-1807-47fd-a147-af28bae5cf12-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.260612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.260527 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff8111-1807-47fd-a147-af28bae5cf12-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.260612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.260542 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjm7x\" (UniqueName: \"kubernetes.io/projected/5b2135eb-5813-4588-ac68-317b47fca015-kube-api-access-vjm7x\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.260612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.260558 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b2135eb-5813-4588-ac68-317b47fca015-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.260612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.260571 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8smdw\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-kube-api-access-8smdw\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.260612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.260583 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dff8111-1807-47fd-a147-af28bae5cf12-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.273479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.273394 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:10.548655 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.548615 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp"] Nov 03 13:35:10.559936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.559893 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-7q5fp"] Nov 03 13:35:10.952175 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:10.952097 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" path="/var/lib/kubelet/pods/7e2cca85-c79f-4f8b-9852-06ea26c6f2fa/volumes" Nov 03 13:35:11.026198 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.026162 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" Nov 03 13:35:11.026396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.026221 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8" event={"ID":"5b2135eb-5813-4588-ac68-317b47fca015","Type":"ContainerDied","Data":"0dc51deea33f2d462ef34197ea18beb2eceed2cadd204111997ba11480c5d4a5"} Nov 03 13:35:11.026396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.026287 4924 scope.go:122] "RemoveContainer" containerID="07f1642436acdd8855e4252564d5f6d88fc767a4d41f8a44076fbb8a8f08cb75" Nov 03 13:35:11.028044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.028010 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" event={"ID":"7dff8111-1807-47fd-a147-af28bae5cf12","Type":"ContainerDied","Data":"8b0f84aaa8a74893fafb20e39ec5f9c79b5bacc3c260f7dc0b856cfeaf216e5d"} Nov 03 13:35:11.028044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.028024 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7" Nov 03 13:35:11.036586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.036567 4924 scope.go:122] "RemoveContainer" containerID="90b97bff2bbca93cb53c5c1b49a52ba5ac887d3986cd407b9d23de99d8379888" Nov 03 13:35:11.185759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.185721 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7"] Nov 03 13:35:11.232172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.232065 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-74sk7"] Nov 03 13:35:11.291523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.291488 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8"] Nov 03 13:35:11.297766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:11.297736 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-9gnn8"] Nov 03 13:35:12.954392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:12.954348 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b2135eb-5813-4588-ac68-317b47fca015" path="/var/lib/kubelet/pods/5b2135eb-5813-4588-ac68-317b47fca015/volumes" Nov 03 13:35:12.955134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:12.955114 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" path="/var/lib/kubelet/pods/7dff8111-1807-47fd-a147-af28bae5cf12/volumes" Nov 03 13:35:14.041303 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.041268 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:14.612537 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.612496 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:14.883399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.883009 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:35:14.921115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.920981 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-data\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921080 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-credential-socket\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921112 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-envoy\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921194 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-socket\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921222 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-certs\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921274 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68f6071-52f8-45ac-9aac-6a917842cdda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68f6071-52f8-45ac-9aac-6a917842cdda-istiod-ca-cert\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921351 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-podinfo\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921396 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-token\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-token\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921391 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-credential-socket" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:14.921744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921435 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-data" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:14.921744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921476 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-kube-api-access-5jbfs\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-kube-api-access-5jbfs\") pod \"c68f6071-52f8-45ac-9aac-6a917842cdda\" (UID: \"c68f6071-52f8-45ac-9aac-6a917842cdda\") " Nov 03 13:35:14.921744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921537 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-socket" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:14.922091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.921785 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68f6071-52f8-45ac-9aac-6a917842cdda-istiod-ca-cert" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:14.922171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.922147 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-certs" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:14.922300 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.922262 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:14.922300 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.922287 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:14.922523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.922300 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68f6071-52f8-45ac-9aac-6a917842cdda-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:14.922523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.922314 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:14.924231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.924207 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-kube-api-access-5jbfs" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "kube-api-access-5jbfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:14.924333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.924289 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-token" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:14.924389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.924371 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-podinfo" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:14.924832 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:14.924788 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-envoy" pod "c68f6071-52f8-45ac-9aac-6a917842cdda" (UID: "c68f6071-52f8-45ac-9aac-6a917842cdda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:15.024050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.024012 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:15.024050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.024049 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:15.024050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.024064 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jbfs\" (UniqueName: \"kubernetes.io/projected/c68f6071-52f8-45ac-9aac-6a917842cdda-kube-api-access-5jbfs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:15.024050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.024096 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:15.024462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.024110 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68f6071-52f8-45ac-9aac-6a917842cdda-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:15.110153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.106013 4924 generic.go:356] "Generic (PLEG): container finished" podID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerID="40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff" exitCode=0 Nov 03 13:35:15.110153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.106295 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" event={"ID":"c68f6071-52f8-45ac-9aac-6a917842cdda","Type":"ContainerDied","Data":"40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff"} Nov 03 13:35:15.110153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.106357 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" event={"ID":"c68f6071-52f8-45ac-9aac-6a917842cdda","Type":"ContainerDied","Data":"1715c16f74e9659cb786000ba1bb6755e5434717579ffb04e6d79e20738ae7cf"} Nov 03 13:35:15.110153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.106386 4924 scope.go:122] "RemoveContainer" containerID="40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff" Nov 03 13:35:15.110153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.106549 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2" Nov 03 13:35:15.183400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.183277 4924 scope.go:122] "RemoveContainer" containerID="40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff" Nov 03 13:35:15.183641 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:35:15.183611 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff\": container with ID starting with 40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff not found: ID does not exist" containerID="40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff" Nov 03 13:35:15.183728 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.183655 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff"} err="failed to get container status \"40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff\": rpc error: code = NotFound desc = could not find container \"40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff\": container with ID starting with 40b46d130007304493150e212e123c6ba8e364add879c165599de2911efad0ff not found: ID does not exist" Nov 03 13:35:15.287965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.287930 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2"] Nov 03 13:35:15.292544 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:15.292511 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vpkz2"] Nov 03 13:35:16.954810 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:16.954774 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" path="/var/lib/kubelet/pods/c68f6071-52f8-45ac-9aac-6a917842cdda/volumes" Nov 03 13:35:17.596616 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:17.596585 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:18.559156 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:35:18.559115 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:35:22.805311 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:22.805274 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:23.034262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.034228 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.680381 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn"] Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681257 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681278 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681301 4924 cpu_manager.go:414] "containerMap: removing container" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681309 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681336 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681345 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681370 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a2935bad-4728-4a81-8a5a-1bb52e55099d" containerName="maas-api" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681378 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2935bad-4728-4a81-8a5a-1bb52e55099d" containerName="maas-api" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681401 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681410 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681421 4924 cpu_manager.go:414] "containerMap: removing container" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681428 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681443 4924 cpu_manager.go:414] "containerMap: removing container" podUID="0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" containerName="maas-api" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681450 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" containerName="maas-api" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681469 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681476 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681485 4924 cpu_manager.go:414] "containerMap: removing container" podUID="9542ec83-268f-48d4-a677-4b42c3ac9c5f" containerName="main" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681492 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9542ec83-268f-48d4-a677-4b42c3ac9c5f" containerName="main" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681506 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681513 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681626 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2935bad-4728-4a81-8a5a-1bb52e55099d" containerName="maas-api" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681641 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681650 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b2135eb-5813-4588-ac68-317b47fca015" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681662 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c36e1e6-f31e-4543-87ca-9f96d2d0e6a7" containerName="maas-api" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681672 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68f6071-52f8-45ac-9aac-6a917842cdda" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681681 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9542ec83-268f-48d4-a677-4b42c3ac9c5f" containerName="main" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681692 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8067e6f-06cc-48c5-8e3e-af27a06a8f87" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681703 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="956c58e2-4740-418d-8bdc-16ed0155d895" containerName="istio-proxy" Nov 03 13:35:23.684100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.681711 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2cca85-c79f-4f8b-9852-06ea26c6f2fa" containerName="istio-proxy" Nov 03 13:35:23.694109 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.690647 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736551 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-socket\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736604 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a5ca16-634c-4334-a8fd-3263acaee476-istiod-ca-cert\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736630 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-envoy\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736678 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rx9\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-kube-api-access-n6rx9\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736706 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-credential-socket\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736739 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-certs\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736777 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-podinfo\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736840 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-data\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.736916 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-token\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.758601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.745252 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a770b9-openshift-default-dockercfg-clwk4\"" Nov 03 13:35:23.813234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.813027 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn"] Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848287 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-podinfo\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848394 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-data\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848490 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-token\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848594 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-socket\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848639 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a5ca16-634c-4334-a8fd-3263acaee476-istiod-ca-cert\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848666 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-envoy\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848719 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rx9\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-kube-api-access-n6rx9\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848749 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-credential-socket\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.848794 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-certs\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.849354 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-certs\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.850459 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a5ca16-634c-4334-a8fd-3263acaee476-istiod-ca-cert\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.850752 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-data\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.852159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.851533 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-socket\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.860917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.860620 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-envoy\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.860917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.860878 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-credential-socket\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.864403 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.864281 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:23.910356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.910199 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rx9\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-kube-api-access-n6rx9\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.912198 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.912167 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-token\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:23.916941 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:23.916897 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-podinfo\") pod \"e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:24.038181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:24.038089 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:24.064257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:24.064219 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:24.512053 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:35:24.512019 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a5ca16_634c_4334_a8fd_3263acaee476.slice/crio-e6e680d89dbed0d8ac90538ad7409e17788fe6b09add6e978c169f1ec0a4e6ba WatchSource:0}: Error finding container e6e680d89dbed0d8ac90538ad7409e17788fe6b09add6e978c169f1ec0a4e6ba: Status 404 returned error can't find the container with id e6e680d89dbed0d8ac90538ad7409e17788fe6b09add6e978c169f1ec0a4e6ba Nov 03 13:35:24.513021 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:24.512991 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn"] Nov 03 13:35:24.514629 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:24.514595 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:35:24.514704 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:24.514660 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:35:24.514704 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:24.514691 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:35:25.189614 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:25.189581 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" event={"ID":"a6a5ca16-634c-4334-a8fd-3263acaee476","Type":"ContainerStarted","Data":"d7f45b16403a7fbf28b73d161a7670ea06c2c854d37d01a9f6be729fde3304aa"} Nov 03 13:35:25.190009 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:25.189638 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" event={"ID":"a6a5ca16-634c-4334-a8fd-3263acaee476","Type":"ContainerStarted","Data":"e6e680d89dbed0d8ac90538ad7409e17788fe6b09add6e978c169f1ec0a4e6ba"} Nov 03 13:35:25.304568 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:25.304217 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" podStartSLOduration=2.304198327 podStartE2EDuration="2.304198327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:25.303160479 +0000 UTC m=+1858.497564073" watchObservedRunningTime="2026-11-03 13:35:25.304198327 +0000 UTC m=+1858.498601921" Nov 03 13:35:26.064417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:26.064384 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:26.069945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:26.069918 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:26.198580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:26.198550 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:26.199650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:26.199630 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:35:27.020271 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:27.020239 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:27.390266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:27.390186 4924 scope.go:122] "RemoveContainer" containerID="8327511c85da0e96f1ca8aa8e1a049edccaf2af070466c4bf188d905a6b14ccb" Nov 03 13:35:28.296053 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:28.296024 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:28.471830 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:28.471796 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:28.601158 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:28.598216 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:29.545528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:29.545496 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:29.737985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:29.737955 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:29.905652 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:29.905572 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:32.925611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:32.925560 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:33.097124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:33.092906 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:34.228663 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:34.228623 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:39.917921 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:39.917884 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:45.472403 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:45.472363 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:48.559430 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:35:48.559393 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:35:50.011955 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.011919 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn"] Nov 03 13:35:50.012496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.012237 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" podUID="2ddd6f5e-23e4-4778-909b-79a2ff540375" containerName="maas-api" containerID="cri-o://6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24" gracePeriod=30 Nov 03 13:35:50.106153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.106113 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:50.344494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.344460 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:35:50.361102 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.361055 4924 generic.go:356] "Generic (PLEG): container finished" podID="2ddd6f5e-23e4-4778-909b-79a2ff540375" containerID="6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24" exitCode=0 Nov 03 13:35:50.361325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.361158 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" Nov 03 13:35:50.361325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.361150 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" event={"ID":"2ddd6f5e-23e4-4778-909b-79a2ff540375","Type":"ContainerDied","Data":"6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24"} Nov 03 13:35:50.361325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.361256 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn" event={"ID":"2ddd6f5e-23e4-4778-909b-79a2ff540375","Type":"ContainerDied","Data":"50a81f1e3c2ca4946b8640add483a31b665ce9b11881e838586ca693c4c1857f"} Nov 03 13:35:50.361325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.361305 4924 scope.go:122] "RemoveContainer" containerID="6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24" Nov 03 13:35:50.373850 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.373830 4924 scope.go:122] "RemoveContainer" containerID="6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24" Nov 03 13:35:50.374142 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:35:50.374122 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24\": container with ID starting with 6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24 not found: ID does not exist" containerID="6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24" Nov 03 13:35:50.374209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.374152 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24"} err="failed to get container status \"6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24\": rpc error: code = NotFound desc = could not find container \"6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24\": container with ID starting with 6251ad3feca84a8030025a5adf88dc97c73a2a7e0dc8bf6b1f2f9da895906e24 not found: ID does not exist" Nov 03 13:35:50.514848 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.514806 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ddd6f5e-23e4-4778-909b-79a2ff540375-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddd6f5e-23e4-4778-909b-79a2ff540375-openshift-service-ca\") pod \"2ddd6f5e-23e4-4778-909b-79a2ff540375\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " Nov 03 13:35:50.515260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.514864 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-metrics-certs\") pod \"2ddd6f5e-23e4-4778-909b-79a2ff540375\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " Nov 03 13:35:50.515260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.515193 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls\") pod \"2ddd6f5e-23e4-4778-909b-79a2ff540375\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " Nov 03 13:35:50.515402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.515346 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ddd6f5e-23e4-4778-909b-79a2ff540375-kube-api-access-4z2xn\" (UniqueName: \"kubernetes.io/projected/2ddd6f5e-23e4-4778-909b-79a2ff540375-kube-api-access-4z2xn\") pod \"2ddd6f5e-23e4-4778-909b-79a2ff540375\" (UID: \"2ddd6f5e-23e4-4778-909b-79a2ff540375\") " Nov 03 13:35:50.516887 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.516859 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddd6f5e-23e4-4778-909b-79a2ff540375-openshift-service-ca" pod "2ddd6f5e-23e4-4778-909b-79a2ff540375" (UID: "2ddd6f5e-23e4-4778-909b-79a2ff540375"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:50.522707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.522681 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddd6f5e-23e4-4778-909b-79a2ff540375-kube-api-access-4z2xn" pod "2ddd6f5e-23e4-4778-909b-79a2ff540375" (UID: "2ddd6f5e-23e4-4778-909b-79a2ff540375"). InnerVolumeSpecName "kube-api-access-4z2xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:50.523187 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.523165 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-metrics-certs" pod "2ddd6f5e-23e4-4778-909b-79a2ff540375" (UID: "2ddd6f5e-23e4-4778-909b-79a2ff540375"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:35:50.535976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.535944 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls" pod "2ddd6f5e-23e4-4778-909b-79a2ff540375" (UID: "2ddd6f5e-23e4-4778-909b-79a2ff540375"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:35:50.616744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.616642 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddd6f5e-23e4-4778-909b-79a2ff540375-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:50.616744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.616676 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:50.616744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.616685 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddd6f5e-23e4-4778-909b-79a2ff540375-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:50.616744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.616694 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2xn\" (UniqueName: \"kubernetes.io/projected/2ddd6f5e-23e4-4778-909b-79a2ff540375-kube-api-access-4z2xn\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:50.780840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.780799 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn"] Nov 03 13:35:50.786483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.786448 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7ca74-84554b7544-qd4hn"] Nov 03 13:35:50.952485 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:50.952447 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ddd6f5e-23e4-4778-909b-79a2ff540375" path="/var/lib/kubelet/pods/2ddd6f5e-23e4-4778-909b-79a2ff540375/volumes" Nov 03 13:35:52.128598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:52.128557 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:56.959350 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:56.959317 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:57.098242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:57.098210 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:58.610376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:58.610314 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:59.183174 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:59.183129 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:35:59.388100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:35:59.385191 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:00.121944 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.121910 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:00.688917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.688877 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2"] Nov 03 13:36:00.689466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.689450 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2ddd6f5e-23e4-4778-909b-79a2ff540375" containerName="maas-api" Nov 03 13:36:00.689515 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.689468 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddd6f5e-23e4-4778-909b-79a2ff540375" containerName="maas-api" Nov 03 13:36:00.689568 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.689558 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddd6f5e-23e4-4778-909b-79a2ff540375" containerName="maas-api" Nov 03 13:36:00.689602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.689569 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dff8111-1807-47fd-a147-af28bae5cf12" containerName="istio-proxy" Nov 03 13:36:00.694899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.694872 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.755101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.753429 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-02f3a0-openshift-default-dockercfg-czfpr\"" Nov 03 13:36:00.760975 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.760933 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-envoy\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.761350 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.761323 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77521ec-beb3-4383-b2e2-e2bb66557174-istiod-ca-cert\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.761528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.761512 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-data\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.761647 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.761633 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-socket\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.761773 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.761757 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8m9r\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-kube-api-access-n8m9r\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.761898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.761883 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-certs\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.762062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.762047 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-podinfo\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.762252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.762235 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-token\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.762359 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.762347 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-credential-socket\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.835279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.834604 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2"] Nov 03 13:36:00.836997 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.836968 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:00.871906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.871863 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-envoy\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.872284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.872257 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77521ec-beb3-4383-b2e2-e2bb66557174-istiod-ca-cert\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.872510 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.872493 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-data\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.872625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.872611 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-socket\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.872724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.872712 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8m9r\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-kube-api-access-n8m9r\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.872845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.872833 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-certs\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.873039 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.873022 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-podinfo\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.873181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.873162 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-token\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.873302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.873286 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-credential-socket\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.873933 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.873912 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-credential-socket\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.878865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.876766 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-envoy\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.878865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.877428 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77521ec-beb3-4383-b2e2-e2bb66557174-istiod-ca-cert\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.878865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.877665 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-data\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.878865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.877896 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-socket\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.878865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.878472 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-certs\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.925092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.908387 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-podinfo\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.952678 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.951725 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8m9r\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-kube-api-access-n8m9r\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:00.954600 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:00.954570 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-token\") pod \"e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:01.016879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:01.016847 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:01.198039 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:01.197915 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:01.458741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:01.458706 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2"] Nov 03 13:36:01.489738 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:36:01.489697 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77521ec_beb3_4383_b2e2_e2bb66557174.slice/crio-9b247fd34eccd260c84bdb67a9938703127889a86b01228fb7b60fbe26bca58f WatchSource:0}: Error finding container 9b247fd34eccd260c84bdb67a9938703127889a86b01228fb7b60fbe26bca58f: Status 404 returned error can't find the container with id 9b247fd34eccd260c84bdb67a9938703127889a86b01228fb7b60fbe26bca58f Nov 03 13:36:01.492179 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:01.492139 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:36:01.492266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:01.492224 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:36:01.492266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:01.492256 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:36:02.450505 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:02.450467 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" event={"ID":"d77521ec-beb3-4383-b2e2-e2bb66557174","Type":"ContainerStarted","Data":"09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2"} Nov 03 13:36:02.450931 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:02.450547 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" event={"ID":"d77521ec-beb3-4383-b2e2-e2bb66557174","Type":"ContainerStarted","Data":"9b247fd34eccd260c84bdb67a9938703127889a86b01228fb7b60fbe26bca58f"} Nov 03 13:36:02.563761 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:02.563689 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" podStartSLOduration=2.563669012 podStartE2EDuration="2.563669012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:02.562278403 +0000 UTC m=+1895.756681993" watchObservedRunningTime="2026-11-03 13:36:02.563669012 +0000 UTC m=+1895.758072606" Nov 03 13:36:03.021289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:03.019741 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:03.029923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:03.028913 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:03.457130 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:03.457089 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:03.458060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:03.458041 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:36:05.222454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:05.222413 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:06.291694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:06.291651 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:06.479591 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:06.479558 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:06.613201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:06.613119 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:08.410722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:08.410678 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:09.337424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:09.337377 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:09.974091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:09.961867 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:10.262951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:10.262838 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:10.550958 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:10.550865 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:10.749564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:10.749507 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:11.246989 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:11.246913 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:12.181879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:12.181840 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:12.329663 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:12.329624 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:16.185019 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:16.184730 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:16.358220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:16.358185 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:16.510325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:16.510234 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:16.974782 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:16.974744 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:17.239680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:17.239582 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:18.559261 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:36:18.559224 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:36:20.536373 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:20.536341 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:20.735239 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:20.735195 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:22.176832 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:22.176779 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:22.938799 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:22.938763 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:25.021056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:25.021020 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:25.169031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:25.168995 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:26.812284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:26.812250 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:26.944651 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:26.944612 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:27.370528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:27.370491 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:30.688807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:30.688773 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:31.335280 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:31.335242 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:31.497289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:31.497246 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:31.660129 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:31.660095 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:35.938514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:35.938471 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc"] Nov 03 13:36:35.943811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:35.943786 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:35.992967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:35.992938 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a770b9-serving-cert\"" Nov 03 13:36:36.045252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.045204 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc"] Nov 03 13:36:36.049224 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.049197 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.049384 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.049348 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7846be-e8bc-4306-aad2-ba4c17850dda-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.049384 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.049370 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-metrics-certs\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.049501 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.049391 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqc2p\" (UniqueName: \"kubernetes.io/projected/ae7846be-e8bc-4306-aad2-ba4c17850dda-kube-api-access-xqc2p\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.112499 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.112460 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:36.150680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.150643 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7846be-e8bc-4306-aad2-ba4c17850dda-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.150680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.150683 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-metrics-certs\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.150949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.150706 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc2p\" (UniqueName: \"kubernetes.io/projected/ae7846be-e8bc-4306-aad2-ba4c17850dda-kube-api-access-xqc2p\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.150949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.150817 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.150949 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:36:36.150919 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-serving-cert: secret "maas-api-e2e-ipp-a-a770b9-serving-cert" not found Nov 03 13:36:36.151123 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:36:36.151000 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls podName:ae7846be-e8bc-4306-aad2-ba4c17850dda nodeName:}" failed. No retries permitted until 2026-11-03 13:36:36.650978307 +0000 UTC m=+1929.845381896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls") pod "maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" (UID: "ae7846be-e8bc-4306-aad2-ba4c17850dda") : secret "maas-api-e2e-ipp-a-a770b9-serving-cert" not found Nov 03 13:36:36.151466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.151445 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7846be-e8bc-4306-aad2-ba4c17850dda-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.153148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.153127 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-metrics-certs\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.206585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.206502 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc2p\" (UniqueName: \"kubernetes.io/projected/ae7846be-e8bc-4306-aad2-ba4c17850dda-kube-api-access-xqc2p\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.278282 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.278245 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:36.420059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.419569 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:36.656807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.656775 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.659211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.659188 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:36.861435 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:36.861383 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:37.232108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:37.232061 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc"] Nov 03 13:36:37.232697 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:36:37.232661 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7846be_e8bc_4306_aad2_ba4c17850dda.slice/crio-a08e68cfb6ca6184f53e885ddca488ba7a66282c081eb9c32f9866da6c9289aa WatchSource:0}: Error finding container a08e68cfb6ca6184f53e885ddca488ba7a66282c081eb9c32f9866da6c9289aa: Status 404 returned error can't find the container with id a08e68cfb6ca6184f53e885ddca488ba7a66282c081eb9c32f9866da6c9289aa Nov 03 13:36:37.234057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:37.234037 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:36:37.698843 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:37.698806 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" event={"ID":"ae7846be-e8bc-4306-aad2-ba4c17850dda","Type":"ContainerStarted","Data":"a08e68cfb6ca6184f53e885ddca488ba7a66282c081eb9c32f9866da6c9289aa"} Nov 03 13:36:38.706680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:38.706641 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" event={"ID":"ae7846be-e8bc-4306-aad2-ba4c17850dda","Type":"ContainerStarted","Data":"312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e"} Nov 03 13:36:38.707112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:38.706746 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:38.817401 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:38.817344 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" podStartSLOduration=2.875466549 podStartE2EDuration="3.817329634s" totalImagesPullingTime="941.863085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:38.815343301 +0000 UTC m=+1932.009746900" watchObservedRunningTime="2026-11-03 13:36:38.817329634 +0000 UTC m=+1932.011733228" Nov 03 13:36:41.500131 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:41.500099 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:42.400159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:42.400127 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:42.548495 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:42.548452 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:44.718548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:44.718518 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:36:46.774233 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:46.774197 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:48.559345 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:36:48.559302 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:36:51.686085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:51.686040 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:51.813244 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:51.813213 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:51.940962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:51.940868 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:52.201717 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:52.201615 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:36:57.460710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:36:57.460673 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:01.982052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:01.982017 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:02.520026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:02.519976 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:02.684139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:02.684104 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:04.125194 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:04.125164 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:04.256097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:04.255976 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:04.413241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:04.413210 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:05.904402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:05.904362 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:06.467851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:06.467819 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:07.292832 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:07.292801 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:08.094925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:08.094896 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:08.666394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:08.666360 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:09.944085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:09.944037 4924 generic.go:356] "Generic (PLEG): container finished" podID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerID="d7f45b16403a7fbf28b73d161a7670ea06c2c854d37d01a9f6be729fde3304aa" exitCode=0 Nov 03 13:37:09.944510 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:09.944128 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" event={"ID":"a6a5ca16-634c-4334-a8fd-3263acaee476","Type":"ContainerDied","Data":"d7f45b16403a7fbf28b73d161a7670ea06c2c854d37d01a9f6be729fde3304aa"} Nov 03 13:37:09.944595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:09.944574 4924 scope.go:122] "RemoveContainer" containerID="d7f45b16403a7fbf28b73d161a7670ea06c2c854d37d01a9f6be729fde3304aa" Nov 03 13:37:09.946136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:09.946096 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:09.946247 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:09.946176 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:09.946247 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:09.946209 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:10.970133 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:10.970097 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" event={"ID":"a6a5ca16-634c-4334-a8fd-3263acaee476","Type":"ContainerStarted","Data":"ca06fe0e2a61a05d6d6a65cd0f361efa9df535d7e71f6be515370c2b8cfc98aa"} Nov 03 13:37:11.064452 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:11.064406 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:37:11.065637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:11.065608 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:37:11.065789 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:11.065697 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:37:11.199166 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:11.199130 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:37:11.719394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:11.719358 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:11.870874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:11.870840 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:11.995943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:11.995828 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:12.072096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:12.065188 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:37:12.072096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:12.065281 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:37:13.064769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:13.064725 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:37:13.065311 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:13.064792 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:37:13.538096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:13.532544 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:14.067149 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:14.067112 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:37:14.067762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:14.067733 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:37:15.067729 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:15.067692 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:37:15.068344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:15.068309 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:37:16.064553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:16.064514 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:37:16.064765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:16.064577 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:37:17.069284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:17.069249 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:37:17.070452 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:17.070422 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:37:17.477540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:17.477503 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:18.558389 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:37:18.558349 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:37:20.667296 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:20.667262 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc"] Nov 03 13:37:20.667841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:20.667456 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" containerID="cri-o://d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8" gracePeriod=30 Nov 03 13:37:20.713990 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:20.713953 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb"] Nov 03 13:37:20.714254 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:20.714228 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" containerID="cri-o://f106b12c01ccf79ceca11f23c86fbeeeefa2dfed8a8d1f82c968ac782ad31a55" gracePeriod=30 Nov 03 13:37:20.717628 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:20.717605 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74"] Nov 03 13:37:20.717796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:20.717775 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" containerID="cri-o://c0c1b6ba95de41512352612a9d33983b9d33f44ae453c79e14caf39fc21367fa" gracePeriod=30 Nov 03 13:37:20.876912 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:20.876877 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-54t74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:37:20.877111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:20.876941 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:37:22.289223 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:22.289185 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:22.709692 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:22.709654 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:37:22.709869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:22.709735 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:37:22.864740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:22.864704 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-86scc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:37:22.864931 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:22.864770 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:37:24.017519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:24.017487 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:24.182346 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:24.182309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:24.311488 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:24.311405 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:26.059881 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.059540 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:37:26.101551 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.101248 4924 generic.go:356] "Generic (PLEG): container finished" podID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerID="f106b12c01ccf79ceca11f23c86fbeeeefa2dfed8a8d1f82c968ac782ad31a55" exitCode=0 Nov 03 13:37:26.101551 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.101487 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" event={"ID":"2b31c398-2145-4e13-a73e-5715bdce2ae4","Type":"ContainerDied","Data":"f106b12c01ccf79ceca11f23c86fbeeeefa2dfed8a8d1f82c968ac782ad31a55"} Nov 03 13:37:26.104585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.104556 4924 generic.go:356] "Generic (PLEG): container finished" podID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerID="c0c1b6ba95de41512352612a9d33983b9d33f44ae453c79e14caf39fc21367fa" exitCode=0 Nov 03 13:37:26.104973 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.104927 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" event={"ID":"a9a879b5-cce0-4eba-862b-dfbd25bb7d12","Type":"ContainerDied","Data":"c0c1b6ba95de41512352612a9d33983b9d33f44ae453c79e14caf39fc21367fa"} Nov 03 13:37:26.111717 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.110762 4924 generic.go:356] "Generic (PLEG): container finished" podID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerID="d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8" exitCode=0 Nov 03 13:37:26.112289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.112262 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" Nov 03 13:37:26.112482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.111967 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" event={"ID":"f6979f54-42fc-4c34-9a00-f718a1d04297","Type":"ContainerDied","Data":"d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8"} Nov 03 13:37:26.112562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.112515 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc" event={"ID":"f6979f54-42fc-4c34-9a00-f718a1d04297","Type":"ContainerDied","Data":"978239a19b8485da45b0664d8dda6ec0953912c1936f2c555b8366c00bf32966"} Nov 03 13:37:26.112562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.112551 4924 scope.go:122] "RemoveContainer" containerID="d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8" Nov 03 13:37:26.119552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.118399 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:37:26.128281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.128262 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:37:26.142159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.141634 4924 scope.go:122] "RemoveContainer" containerID="d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8" Nov 03 13:37:26.142159 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:37:26.142023 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8\": container with ID starting with d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8 not found: ID does not exist" containerID="d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8" Nov 03 13:37:26.142159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.142064 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8"} err="failed to get container status \"d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8\": rpc error: code = NotFound desc = could not find container \"d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8\": container with ID starting with d25c7a1b680874b94f405c77e5030d131aa8393c7e40f0172239a13cbae17be8 not found: ID does not exist" Nov 03 13:37:26.193507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193413 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-envoy\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.193507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193456 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-data\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.193507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193477 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-socket\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.193794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193534 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-envoy\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.193794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193562 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-podinfo\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.193794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193600 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-token\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-token\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.193794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193641 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istiod-ca-cert\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.193794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193667 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-data\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.193794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193722 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-data\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.193794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193769 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-certs\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.193794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193780 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-socket" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193796 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-podinfo\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193797 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-data" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193831 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6979f54-42fc-4c34-9a00-f718a1d04297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6979f54-42fc-4c34-9a00-f718a1d04297-istiod-ca-cert\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193871 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b31c398-2145-4e13-a73e-5715bdce2ae4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b31c398-2145-4e13-a73e-5715bdce2ae4-istiod-ca-cert\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193921 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-token\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-token\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.193979 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-envoy\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194023 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-token\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-token\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194083 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istiod-ca-cert" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194108 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-podinfo\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.194178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194152 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-kube-api-access-hhwm4\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-kube-api-access-hhwm4\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194198 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-credential-socket\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194235 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-certs\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194272 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-kube-api-access-wqd7m\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-kube-api-access-wqd7m\") pod \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\" (UID: \"a9a879b5-cce0-4eba-862b-dfbd25bb7d12\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194301 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-socket\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194358 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-credential-socket\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194394 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-kube-api-access-h7lgc\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-kube-api-access-h7lgc\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194443 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-certs\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194479 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-socket\") pod \"f6979f54-42fc-4c34-9a00-f718a1d04297\" (UID: \"f6979f54-42fc-4c34-9a00-f718a1d04297\") " Nov 03 13:37:26.194622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.194541 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-credential-socket\") pod \"2b31c398-2145-4e13-a73e-5715bdce2ae4\" (UID: \"2b31c398-2145-4e13-a73e-5715bdce2ae4\") " Nov 03 13:37:26.195232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.195206 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.195346 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.195235 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.195346 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.195249 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.195446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.195420 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-credential-socket" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.195627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.195604 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-certs" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.196125 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.195722 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b31c398-2145-4e13-a73e-5715bdce2ae4-istiod-ca-cert" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:26.196125 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.195814 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-socket" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.196125 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.195896 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6979f54-42fc-4c34-9a00-f718a1d04297-istiod-ca-cert" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:26.196125 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.196014 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-credential-socket" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.196417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.196189 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-socket" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.196886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.196861 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-data" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.196982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.196953 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-certs" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.197149 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.197049 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-data" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.197149 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.197111 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-envoy" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.198085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.197293 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-certs" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.198085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.197368 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-podinfo" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:26.198085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.197480 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-credential-socket" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.200552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.200526 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-kube-api-access-hhwm4" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "kube-api-access-hhwm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:26.200666 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.200564 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-kube-api-access-wqd7m" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "kube-api-access-wqd7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:26.200666 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.200632 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-token" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:26.200826 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.200798 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-kube-api-access-h7lgc" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "kube-api-access-h7lgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:26.200943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.200921 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-token" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:26.201089 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.201033 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-envoy" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.201190 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.201115 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-podinfo" pod "a9a879b5-cce0-4eba-862b-dfbd25bb7d12" (UID: "a9a879b5-cce0-4eba-862b-dfbd25bb7d12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:26.201432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.201415 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-podinfo" pod "f6979f54-42fc-4c34-9a00-f718a1d04297" (UID: "f6979f54-42fc-4c34-9a00-f718a1d04297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:26.201721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.201688 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-envoy" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:26.202148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.202127 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-token" pod "2b31c398-2145-4e13-a73e-5715bdce2ae4" (UID: "2b31c398-2145-4e13-a73e-5715bdce2ae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:26.296032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.295987 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296021 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296034 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296043 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296052 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296061 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296085 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296098 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296111 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296125 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296138 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296149 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6979f54-42fc-4c34-9a00-f718a1d04297-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296157 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b31c398-2145-4e13-a73e-5715bdce2ae4-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296165 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296174 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296182 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296190 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6979f54-42fc-4c34-9a00-f718a1d04297-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296200 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhwm4\" (UniqueName: \"kubernetes.io/projected/2b31c398-2145-4e13-a73e-5715bdce2ae4-kube-api-access-hhwm4\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296211 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296219 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296227 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqd7m\" (UniqueName: \"kubernetes.io/projected/a9a879b5-cce0-4eba-862b-dfbd25bb7d12-kube-api-access-wqd7m\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296235 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b31c398-2145-4e13-a73e-5715bdce2ae4-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296244 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6979f54-42fc-4c34-9a00-f718a1d04297-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.296288 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.296260 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7lgc\" (UniqueName: \"kubernetes.io/projected/f6979f54-42fc-4c34-9a00-f718a1d04297-kube-api-access-h7lgc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:26.614061 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.613950 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc"] Nov 03 13:37:26.617658 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.617618 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-86scc"] Nov 03 13:37:26.953744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:26.953708 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" path="/var/lib/kubelet/pods/f6979f54-42fc-4c34-9a00-f718a1d04297/volumes" Nov 03 13:37:27.119858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.119824 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" Nov 03 13:37:27.119858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.119835 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74" event={"ID":"a9a879b5-cce0-4eba-862b-dfbd25bb7d12","Type":"ContainerDied","Data":"3d0a3637f0a36505c15fd77c61a358f5e29c3d19d1635e4cf5842c10ea020f75"} Nov 03 13:37:27.120459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.119900 4924 scope.go:122] "RemoveContainer" containerID="c0c1b6ba95de41512352612a9d33983b9d33f44ae453c79e14caf39fc21367fa" Nov 03 13:37:27.123685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.123660 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" Nov 03 13:37:27.123818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.123736 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb" event={"ID":"2b31c398-2145-4e13-a73e-5715bdce2ae4","Type":"ContainerDied","Data":"92fe56a364fb2c2d41f3cb20c6ac68019775fad06cf01edf3dc4b623841966c0"} Nov 03 13:37:27.131793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.131770 4924 scope.go:122] "RemoveContainer" containerID="f106b12c01ccf79ceca11f23c86fbeeeefa2dfed8a8d1f82c968ac782ad31a55" Nov 03 13:37:27.277004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.276925 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74"] Nov 03 13:37:27.323057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.323023 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-54t74"] Nov 03 13:37:27.383972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.383937 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb"] Nov 03 13:37:27.388948 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.388921 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j6nfb"] Nov 03 13:37:27.696136 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:27.696065 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:28.448534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:28.448497 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:28.611689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:28.611652 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:28.951956 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:28.951923 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" path="/var/lib/kubelet/pods/2b31c398-2145-4e13-a73e-5715bdce2ae4/volumes" Nov 03 13:37:28.952407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:28.952392 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" path="/var/lib/kubelet/pods/a9a879b5-cce0-4eba-862b-dfbd25bb7d12/volumes" Nov 03 13:37:29.656954 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:29.656913 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:30.753400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:30.753360 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:31.310056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:31.310020 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:31.466095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:31.463242 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2"] Nov 03 13:37:31.466095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:31.463474 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" containerID="cri-o://a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735" gracePeriod=30 Nov 03 13:37:31.538769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:31.538730 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l"] Nov 03 13:37:31.539020 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:31.538987 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" containerID="cri-o://aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77" gracePeriod=30 Nov 03 13:37:32.814384 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:32.814349 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:33.374313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374277 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w"] Nov 03 13:37:33.374859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374842 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" Nov 03 13:37:33.374859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374859 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" Nov 03 13:37:33.375015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374878 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" Nov 03 13:37:33.375015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374884 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" Nov 03 13:37:33.375015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374895 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" Nov 03 13:37:33.375015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374900 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" Nov 03 13:37:33.375015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374974 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b31c398-2145-4e13-a73e-5715bdce2ae4" containerName="istio-proxy" Nov 03 13:37:33.375015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374985 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a879b5-cce0-4eba-862b-dfbd25bb7d12" containerName="istio-proxy" Nov 03 13:37:33.375015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.374992 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6979f54-42fc-4c34-9a00-f718a1d04297" containerName="istio-proxy" Nov 03 13:37:33.380450 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.380427 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.448507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.448468 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-21f111-openshift-default-dockercfg-44kjn\"" Nov 03 13:37:33.454053 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.454019 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:33.510197 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510161 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16c9379-dc86-4ecd-a669-ebe078128f6f-istiod-ca-cert\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.510395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510221 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-certs\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.510395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510254 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cg6l\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-kube-api-access-4cg6l\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.510395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510291 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-token\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.510395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510341 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-data\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.510622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510400 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-socket\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.510622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510480 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-envoy\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.510622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510521 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-credential-socket\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.510772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.510654 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-podinfo\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.536268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.536212 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w"] Nov 03 13:37:33.611868 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.611831 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-podinfo\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.611893 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16c9379-dc86-4ecd-a669-ebe078128f6f-istiod-ca-cert\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.611932 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-certs\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.611960 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cg6l\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-kube-api-access-4cg6l\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612219 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612169 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-token\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612280 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612225 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-data\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612283 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-socket\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612349 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-envoy\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612380 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-credential-socket\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612596 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-data\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612677 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-socket\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612708 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-credential-socket\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612838 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612783 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-certs\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.612873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.612836 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16c9379-dc86-4ecd-a669-ebe078128f6f-istiod-ca-cert\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.615276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.615255 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-envoy\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.615506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.615490 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-podinfo\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.616629 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.616594 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:33.668906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.668861 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cg6l\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-kube-api-access-4cg6l\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.672605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.672536 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-token\") pod \"e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:33.710661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:33.710626 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:34.141482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:34.141452 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w"] Nov 03 13:37:34.142785 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:37:34.142759 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16c9379_dc86_4ecd_a669_ebe078128f6f.slice/crio-bd1fe86647264e2b09b3bf623b382e2182d43b84821c35b51062ae0d67f0aba5 WatchSource:0}: Error finding container bd1fe86647264e2b09b3bf623b382e2182d43b84821c35b51062ae0d67f0aba5: Status 404 returned error can't find the container with id bd1fe86647264e2b09b3bf623b382e2182d43b84821c35b51062ae0d67f0aba5 Nov 03 13:37:34.145870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:34.145835 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:34.145976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:34.145914 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:34.145976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:34.145961 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:34.178917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:34.178877 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" event={"ID":"b16c9379-dc86-4ecd-a669-ebe078128f6f","Type":"ContainerStarted","Data":"bd1fe86647264e2b09b3bf623b382e2182d43b84821c35b51062ae0d67f0aba5"} Nov 03 13:37:35.187685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:35.187627 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" event={"ID":"b16c9379-dc86-4ecd-a669-ebe078128f6f","Type":"ContainerStarted","Data":"a93737a996f930070e509bf077d61c4f764e93e204ef53a23fade321c2b1a3b1"} Nov 03 13:37:35.283153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:35.283110 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:37:35.283315 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:35.283176 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:37:35.710981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:35.710946 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:35.716538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:35.716513 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:35.833104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:35.833030 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podStartSLOduration=2.8330174489999997 podStartE2EDuration="2.833017449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:35.382512021 +0000 UTC m=+1988.576915612" watchObservedRunningTime="2026-11-03 13:37:35.833017449 +0000 UTC m=+1989.027421043" Nov 03 13:37:36.100575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.100492 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:37:36.100575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.100545 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:37:36.194789 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.194753 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:36.195956 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.195930 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:37:36.824233 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.824205 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:37:36.858936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.858909 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:37:36.954675 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954641 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1bb9f30-a784-4180-aa25-64a3e8e79381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb9f30-a784-4180-aa25-64a3e8e79381-istiod-ca-cert\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.954675 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954676 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-istio-token\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-istio-token\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.954920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954745 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-credential-socket\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.954920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954772 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-envoy\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.954920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954807 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-credential-socket\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.954920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954837 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-podinfo\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.954920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954864 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-certs\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.954920 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954902 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-socket\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.955246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.954949 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-socket\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.955246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955022 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-data\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.955246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955063 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337a993b-ad5a-4066-b7b0-417607091a40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a993b-ad5a-4066-b7b0-417607091a40-istiod-ca-cert\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.955246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955093 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bb9f30-a784-4180-aa25-64a3e8e79381-istiod-ca-cert" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:36.955246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955168 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-kube-api-access-dh5dj\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-kube-api-access-dh5dj\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.955246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955202 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-data\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.956178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955254 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-certs\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.956178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955497 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-envoy\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.956178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955538 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-kube-api-access-w2ptr\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-kube-api-access-w2ptr\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.956178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955671 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-token\") pod \"e1bb9f30-a784-4180-aa25-64a3e8e79381\" (UID: \"e1bb9f30-a784-4180-aa25-64a3e8e79381\") " Nov 03 13:37:36.956178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.955703 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/337a993b-ad5a-4066-b7b0-417607091a40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a993b-ad5a-4066-b7b0-417607091a40-istio-podinfo\") pod \"337a993b-ad5a-4066-b7b0-417607091a40\" (UID: \"337a993b-ad5a-4066-b7b0-417607091a40\") " Nov 03 13:37:36.956178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.956160 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-data" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.956496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.956392 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-socket" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.956496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.956404 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-socket" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.956790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.956632 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-certs" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.956790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.956619 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-credential-socket" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.956790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.956646 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-certs" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.957062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.956833 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-data" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.957062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.956886 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.957062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.957009 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1bb9f30-a784-4180-aa25-64a3e8e79381-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.957502 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.957386 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337a993b-ad5a-4066-b7b0-417607091a40-istiod-ca-cert" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:36.958008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.957923 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-istio-token" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:36.958308 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.958280 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-credential-socket" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.958526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.958502 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-podinfo" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:36.967431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.967402 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-envoy" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.967431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.967417 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-envoy" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.967597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.967438 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-token" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:36.967597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.967566 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-kube-api-access-dh5dj" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "kube-api-access-dh5dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:36.967707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.967638 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/337a993b-ad5a-4066-b7b0-417607091a40-istio-podinfo" pod "337a993b-ad5a-4066-b7b0-417607091a40" (UID: "337a993b-ad5a-4066-b7b0-417607091a40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:36.968514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:36.968494 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-kube-api-access-w2ptr" pod "e1bb9f30-a784-4180-aa25-64a3e8e79381" (UID: "e1bb9f30-a784-4180-aa25-64a3e8e79381"). InnerVolumeSpecName "kube-api-access-w2ptr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060100 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060135 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060151 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060164 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060178 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060191 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060203 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060217 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060228 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a993b-ad5a-4066-b7b0-417607091a40-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060241 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh5dj\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-kube-api-access-dh5dj\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060255 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1bb9f30-a784-4180-aa25-64a3e8e79381-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060269 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a993b-ad5a-4066-b7b0-417607091a40-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060283 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2ptr\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-kube-api-access-w2ptr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060296 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1bb9f30-a784-4180-aa25-64a3e8e79381-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060307 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a993b-ad5a-4066-b7b0-417607091a40-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.060842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.060319 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337a993b-ad5a-4066-b7b0-417607091a40-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.128772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.128735 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:37.210631 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.210593 4924 generic.go:356] "Generic (PLEG): container finished" podID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerID="aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77" exitCode=0 Nov 03 13:37:37.211233 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.210677 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" Nov 03 13:37:37.211233 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.210680 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" event={"ID":"e1bb9f30-a784-4180-aa25-64a3e8e79381","Type":"ContainerDied","Data":"aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77"} Nov 03 13:37:37.211233 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.210833 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l" event={"ID":"e1bb9f30-a784-4180-aa25-64a3e8e79381","Type":"ContainerDied","Data":"4b8bcde74a75ea798e808fbe93068a580c7896376498046d13462215d09f8547"} Nov 03 13:37:37.211233 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.210877 4924 scope.go:122] "RemoveContainer" containerID="aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77" Nov 03 13:37:37.214135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.214055 4924 generic.go:356] "Generic (PLEG): container finished" podID="337a993b-ad5a-4066-b7b0-417607091a40" containerID="a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735" exitCode=0 Nov 03 13:37:37.214259 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.214236 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" Nov 03 13:37:37.214337 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.214297 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" event={"ID":"337a993b-ad5a-4066-b7b0-417607091a40","Type":"ContainerDied","Data":"a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735"} Nov 03 13:37:37.214404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.214342 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2" event={"ID":"337a993b-ad5a-4066-b7b0-417607091a40","Type":"ContainerDied","Data":"ada7af3714615e77433ef7441ae579e1984975bfd7b77858bb420fa11d3f4c37"} Nov 03 13:37:37.232317 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.232290 4924 scope.go:122] "RemoveContainer" containerID="aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77" Nov 03 13:37:37.232902 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:37:37.232870 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77\": container with ID starting with aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77 not found: ID does not exist" containerID="aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77" Nov 03 13:37:37.233044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.232916 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77"} err="failed to get container status \"aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77\": rpc error: code = NotFound desc = could not find container \"aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77\": container with ID starting with aa9f4b3e8d3d2e306f3172923dc7c787aa55b3f5e21c11b901293659aea98d77 not found: ID does not exist" Nov 03 13:37:37.233044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.232942 4924 scope.go:122] "RemoveContainer" containerID="a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735" Nov 03 13:37:37.249994 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.249969 4924 scope.go:122] "RemoveContainer" containerID="a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735" Nov 03 13:37:37.250370 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:37:37.250347 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735\": container with ID starting with a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735 not found: ID does not exist" containerID="a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735" Nov 03 13:37:37.250467 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.250377 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735"} err="failed to get container status \"a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735\": rpc error: code = NotFound desc = could not find container \"a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735\": container with ID starting with a340525773a385ad4e2479008b458a5026db8976df2a2389029f2c0422df2735 not found: ID does not exist" Nov 03 13:37:37.368691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.368613 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2"] Nov 03 13:37:37.372680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.372654 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-g4qf2"] Nov 03 13:37:37.477897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.477857 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l"] Nov 03 13:37:37.481024 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:37.480994 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-7z65l"] Nov 03 13:37:38.227567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:38.227534 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:38.414796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:38.414757 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:38.562902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:38.562826 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:38.951320 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:38.951285 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337a993b-ad5a-4066-b7b0-417607091a40" path="/var/lib/kubelet/pods/337a993b-ad5a-4066-b7b0-417607091a40/volumes" Nov 03 13:37:38.951798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:38.951783 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" path="/var/lib/kubelet/pods/e1bb9f30-a784-4180-aa25-64a3e8e79381/volumes" Nov 03 13:37:41.485462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:41.485418 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:41.666015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:41.665978 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:42.803303 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:42.803257 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:47.932224 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:47.932191 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:48.559991 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:37:48.559955 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:37:53.388458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:53.388416 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:56.923850 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:56.923812 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:57.294924 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:37:57.294888 4924 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@4-27835-0.service\": RecentStats: unable to find data in memory cache]" Nov 03 13:37:57.302293 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.302263 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:57.370774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.370739 4924 generic.go:356] "Generic (PLEG): container finished" podID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerID="09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2" exitCode=0 Nov 03 13:37:57.370954 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.370820 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" event={"ID":"d77521ec-beb3-4383-b2e2-e2bb66557174","Type":"ContainerDied","Data":"09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2"} Nov 03 13:37:57.371307 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.371285 4924 scope.go:122] "RemoveContainer" containerID="09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2" Nov 03 13:37:57.373180 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.373139 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:57.373286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.373238 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:57.373338 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.373289 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:37:57.511788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.511709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:57.674459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.674398 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:57.854159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:57.854077 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:37:58.380701 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:58.380664 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" event={"ID":"d77521ec-beb3-4383-b2e2-e2bb66557174","Type":"ContainerStarted","Data":"8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1"} Nov 03 13:37:59.020594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:59.017809 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:37:59.020594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:59.018743 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:37:59.020594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:37:59.018810 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:38:00.017229 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:00.017196 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:38:00.017689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:00.017269 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:38:00.775399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:00.775368 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:01.017418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:01.017380 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:38:01.017834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:01.017436 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:38:02.017334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:02.017291 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:38:02.017541 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:02.017378 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:38:03.018855 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:03.018809 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:38:03.019286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:03.018872 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:38:03.457087 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:03.457035 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:38:04.022447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:04.022420 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:38:04.421138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:04.421099 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:38:06.037088 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:06.037049 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:11.197855 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:11.197772 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:16.220307 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:16.220267 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:16.569850 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:16.569758 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:18.558418 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:38:18.558372 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:38:19.706413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:19.706378 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:19.843671 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:19.843635 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:22.380855 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:22.380821 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:23.596603 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:23.596571 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:28.033793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:28.033755 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:29.259258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:29.259224 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:34.252861 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:34.252823 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:34.400016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:34.399979 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:34.541839 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:34.541762 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:35.015356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:35.015320 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:39.885820 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:39.885781 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69"] Nov 03 13:38:39.886742 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:39.886680 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" podUID="df36687a-57bd-4b91-8b5f-e8f095f700a9" containerName="maas-api" containerID="cri-o://21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046" gracePeriod=30 Nov 03 13:38:39.976359 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:39.976326 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:40.217678 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.217650 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:38:40.326501 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.326453 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls\") pod \"df36687a-57bd-4b91-8b5f-e8f095f700a9\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " Nov 03 13:38:40.326689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.326516 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df36687a-57bd-4b91-8b5f-e8f095f700a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36687a-57bd-4b91-8b5f-e8f095f700a9-openshift-service-ca\") pod \"df36687a-57bd-4b91-8b5f-e8f095f700a9\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " Nov 03 13:38:40.326689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.326627 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df36687a-57bd-4b91-8b5f-e8f095f700a9-kube-api-access-drxn7\" (UniqueName: \"kubernetes.io/projected/df36687a-57bd-4b91-8b5f-e8f095f700a9-kube-api-access-drxn7\") pod \"df36687a-57bd-4b91-8b5f-e8f095f700a9\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " Nov 03 13:38:40.326817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.326715 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-metrics-certs\") pod \"df36687a-57bd-4b91-8b5f-e8f095f700a9\" (UID: \"df36687a-57bd-4b91-8b5f-e8f095f700a9\") " Nov 03 13:38:40.327111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.326980 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df36687a-57bd-4b91-8b5f-e8f095f700a9-openshift-service-ca" pod "df36687a-57bd-4b91-8b5f-e8f095f700a9" (UID: "df36687a-57bd-4b91-8b5f-e8f095f700a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:40.327302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.327278 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36687a-57bd-4b91-8b5f-e8f095f700a9-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:40.329794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.329760 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df36687a-57bd-4b91-8b5f-e8f095f700a9-kube-api-access-drxn7" pod "df36687a-57bd-4b91-8b5f-e8f095f700a9" (UID: "df36687a-57bd-4b91-8b5f-e8f095f700a9"). InnerVolumeSpecName "kube-api-access-drxn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:40.329794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.329765 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-metrics-certs" pod "df36687a-57bd-4b91-8b5f-e8f095f700a9" (UID: "df36687a-57bd-4b91-8b5f-e8f095f700a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:38:40.329942 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.329852 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls" pod "df36687a-57bd-4b91-8b5f-e8f095f700a9" (UID: "df36687a-57bd-4b91-8b5f-e8f095f700a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:38:40.428139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.428064 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:40.428139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.428140 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxn7\" (UniqueName: \"kubernetes.io/projected/df36687a-57bd-4b91-8b5f-e8f095f700a9-kube-api-access-drxn7\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:40.428348 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.428151 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df36687a-57bd-4b91-8b5f-e8f095f700a9-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:40.593697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.593613 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:40.670813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.670777 4924 generic.go:356] "Generic (PLEG): container finished" podID="df36687a-57bd-4b91-8b5f-e8f095f700a9" containerID="21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046" exitCode=0 Nov 03 13:38:40.670996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.670838 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" Nov 03 13:38:40.670996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.670858 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" event={"ID":"df36687a-57bd-4b91-8b5f-e8f095f700a9","Type":"ContainerDied","Data":"21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046"} Nov 03 13:38:40.670996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.670925 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69" event={"ID":"df36687a-57bd-4b91-8b5f-e8f095f700a9","Type":"ContainerDied","Data":"aaabba0d0232336572ef22e9d7e59950e47a15ceb8d30bf34d0e8b338df986c5"} Nov 03 13:38:40.670996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.670949 4924 scope.go:122] "RemoveContainer" containerID="21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046" Nov 03 13:38:40.682266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.682242 4924 scope.go:122] "RemoveContainer" containerID="21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046" Nov 03 13:38:40.682572 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:38:40.682554 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046\": container with ID starting with 21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046 not found: ID does not exist" containerID="21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046" Nov 03 13:38:40.682614 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.682581 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046"} err="failed to get container status \"21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046\": rpc error: code = NotFound desc = could not find container \"21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046\": container with ID starting with 21eaae4350ebba4a2b47e5b0646c89794b2c3c2ab4152c2012b90f93754b8046 not found: ID does not exist" Nov 03 13:38:40.724151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.724117 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:40.827063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.827028 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69"] Nov 03 13:38:40.830961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.830934 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be8d0a-6d6b7c9855-p2b69"] Nov 03 13:38:40.950563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:40.950527 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df36687a-57bd-4b91-8b5f-e8f095f700a9" path="/var/lib/kubelet/pods/df36687a-57bd-4b91-8b5f-e8f095f700a9/volumes" Nov 03 13:38:42.007122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:42.007066 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:42.185507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:42.185472 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:42.305703 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:42.305606 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:44.006015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:44.005979 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:45.047863 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:45.047820 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:45.188278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:45.188241 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:45.756742 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:45.756703 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:46.605592 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:46.605539 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:46.804327 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:46.804289 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:47.976579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:47.976545 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:48.381154 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:48.380690 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:48.567714 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:38:48.567673 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:38:48.894753 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:48.894720 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:49.695574 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:49.695517 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:52.147820 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:52.147784 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:53.339641 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:53.339606 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:55.797607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:55.797573 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:38:59.251332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:38:59.251290 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:03.295747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:03.295716 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:03.849769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:03.849718 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:06.504873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:06.504838 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:07.236251 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.236203 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:39:07.236461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.236278 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:39:07.879005 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.878954 4924 generic.go:356] "Generic (PLEG): container finished" podID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerID="a93737a996f930070e509bf077d61c4f764e93e204ef53a23fade321c2b1a3b1" exitCode=0 Nov 03 13:39:07.879781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.879189 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" event={"ID":"b16c9379-dc86-4ecd-a669-ebe078128f6f","Type":"ContainerDied","Data":"a93737a996f930070e509bf077d61c4f764e93e204ef53a23fade321c2b1a3b1"} Nov 03 13:39:07.886468 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.880391 4924 scope.go:122] "RemoveContainer" containerID="a93737a996f930070e509bf077d61c4f764e93e204ef53a23fade321c2b1a3b1" Nov 03 13:39:07.886468 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.882748 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:07.886468 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.882851 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:07.886468 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.882899 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:07.899112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:07.896917 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:08.483093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:08.482736 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:08.887683 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:08.887599 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" event={"ID":"b16c9379-dc86-4ecd-a669-ebe078128f6f","Type":"ContainerStarted","Data":"39a2119f19c0ceeab811cb9218f4e1d92805832f5995170b05eb7fca5ad46746"} Nov 03 13:39:09.711771 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:09.711734 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:39:09.711975 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:09.711945 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:39:09.712063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:09.711999 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:39:10.711395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:10.711360 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:39:10.711786 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:10.711414 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:39:10.738096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:10.737298 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:10.874546 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:10.874506 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:11.711461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:11.711421 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:39:11.711943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:11.711489 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:39:12.401429 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:12.401393 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:12.711594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:12.711559 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:39:12.712004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:12.711626 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:39:12.969628 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:12.969526 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:13.711319 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:13.711280 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:39:13.711496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:13.711355 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:39:13.901270 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:13.901210 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:14.715657 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:14.715628 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:39:14.716048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:14.716025 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:39:14.717239 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:14.717218 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:39:15.677423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:15.677381 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:16.365448 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:16.346525 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:16.586095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:16.582667 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:18.558843 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:39:18.558803 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:39:19.378272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:19.378234 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:20.451926 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:20.451893 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:20.637856 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:20.637814 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:20.797847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:20.797760 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:21.219518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:21.219475 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:21.411092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:21.410193 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:23.728811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:23.728757 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:23.903096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:23.902932 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:24.673643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:24.673607 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:25.246794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:25.246756 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:25.386359 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:25.386325 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:25.698734 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:25.698702 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:25.932172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:25.932135 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:26.123010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:26.122900 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:27.140400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:27.140344 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:27.280626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:27.280595 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:39:27.282699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:27.282672 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:39:27.498247 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:27.498210 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc"] Nov 03 13:39:27.498451 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:27.498430 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" containerID="cri-o://8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2" gracePeriod=30 Nov 03 13:39:28.392555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:28.392512 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:29.020114 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:29.020064 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:39:29.020283 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:29.020145 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:39:29.184222 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:29.184185 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:29.363841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:29.363732 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:31.254405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:31.254371 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:31.828665 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:31.828631 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:32.761081 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.761052 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:39:32.820918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.820838 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e17ef34d-0799-40fe-b18c-2681d8528771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ef34d-0799-40fe-b18c-2681d8528771-istio-podinfo\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.820918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.820878 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-certs\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.820918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.820913 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-data\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.821221 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.820946 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-socket\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.821221 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.820969 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17ef34d-0799-40fe-b18c-2681d8528771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ef34d-0799-40fe-b18c-2681d8528771-istiod-ca-cert\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.821221 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821044 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-credential-socket\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.821221 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821137 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-kube-api-access-hmtc6\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-kube-api-access-hmtc6\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.821453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821224 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-envoy\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.821453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821277 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-istio-token\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-istio-token\") pod \"e17ef34d-0799-40fe-b18c-2681d8528771\" (UID: \"e17ef34d-0799-40fe-b18c-2681d8528771\") " Nov 03 13:39:32.821626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821273 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-certs" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:32.821626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821285 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-socket" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:32.821626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821316 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-data" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:32.821626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821352 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-credential-socket" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:32.821626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821423 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17ef34d-0799-40fe-b18c-2681d8528771-istiod-ca-cert" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:32.821808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821706 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:32.821808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821724 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ef34d-0799-40fe-b18c-2681d8528771-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:32.821808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821734 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:32.821808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821742 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:32.821808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.821750 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:32.823442 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.823421 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-istio-token" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:32.823573 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.823555 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-envoy" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:32.823621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.823565 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-kube-api-access-hmtc6" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "kube-api-access-hmtc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:32.823621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.823558 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e17ef34d-0799-40fe-b18c-2681d8528771-istio-podinfo" pod "e17ef34d-0799-40fe-b18c-2681d8528771" (UID: "e17ef34d-0799-40fe-b18c-2681d8528771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:39:32.922959 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.922916 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ef34d-0799-40fe-b18c-2681d8528771-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:32.922959 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.922955 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:32.923183 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.922971 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ef34d-0799-40fe-b18c-2681d8528771-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:32.923183 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:32.922986 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmtc6\" (UniqueName: \"kubernetes.io/projected/e17ef34d-0799-40fe-b18c-2681d8528771-kube-api-access-hmtc6\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:33.114532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.114441 4924 generic.go:356] "Generic (PLEG): container finished" podID="e17ef34d-0799-40fe-b18c-2681d8528771" containerID="8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2" exitCode=0 Nov 03 13:39:33.114532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.114493 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" Nov 03 13:39:33.114724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.114524 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" event={"ID":"e17ef34d-0799-40fe-b18c-2681d8528771","Type":"ContainerDied","Data":"8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2"} Nov 03 13:39:33.114724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.114581 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc" event={"ID":"e17ef34d-0799-40fe-b18c-2681d8528771","Type":"ContainerDied","Data":"6802df3f3c6e2bbb687ded5bf482bfa196f62db2fa2dada96c3b142f48427f41"} Nov 03 13:39:33.114724 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.114604 4924 scope.go:122] "RemoveContainer" containerID="8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2" Nov 03 13:39:33.124868 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.124852 4924 scope.go:122] "RemoveContainer" containerID="8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2" Nov 03 13:39:33.125180 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:39:33.125158 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2\": container with ID starting with 8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2 not found: ID does not exist" containerID="8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2" Nov 03 13:39:33.125257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.125193 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2"} err="failed to get container status \"8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2\": rpc error: code = NotFound desc = could not find container \"8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2\": container with ID starting with 8c0e699023a86d5c1ef4ced6c67eb5846f21f637f4e11199f4c7de72343ec6b2 not found: ID does not exist" Nov 03 13:39:33.279550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.279508 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc"] Nov 03 13:39:33.282624 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:33.282598 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6hjpc"] Nov 03 13:39:34.779420 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:34.779385 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:34.952503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:34.952466 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" path="/var/lib/kubelet/pods/e17ef34d-0799-40fe-b18c-2681d8528771/volumes" Nov 03 13:39:34.958146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:34.958118 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:35.222289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:35.222257 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:36.548589 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:36.548543 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:37.396586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:37.396548 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:38.064040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:38.064001 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:41.235565 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:41.235529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:41.687335 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:41.687296 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt"] Nov 03 13:39:41.687598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:41.687572 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" containerID="cri-o://a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc" gracePeriod=30 Nov 03 13:39:42.480707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:42.480670 4924 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:39:42.481125 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:42.480742 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:39:44.316193 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:44.316157 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:45.240446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:45.240410 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:45.871927 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:45.871890 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:46.964043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:46.963982 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:39:47.092804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.092763 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-kube-api-access-mszzv\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-kube-api-access-mszzv\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.092987 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.092877 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istiod-ca-cert\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.093083 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.092983 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-envoy\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.093083 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093046 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-certs\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.093188 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093121 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-credential-socket\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.093257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093191 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-podinfo\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.093312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093254 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-data\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.093369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093311 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-socket\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.093428 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093377 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-credential-socket" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:47.093485 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093427 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istiod-ca-cert" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:47.093609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093584 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-token\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-token\") pod \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\" (UID: \"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15\") " Nov 03 13:39:47.093609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093590 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-certs" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:47.093723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.093618 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-socket" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:47.094334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.094303 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-data" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:47.094708 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.094689 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.094809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.094709 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.094809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.094720 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.094809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.094729 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.094809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.094742 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.095590 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.095564 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-envoy" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:47.096316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.096295 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-podinfo" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:39:47.096900 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.096884 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-token" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:47.097170 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.097156 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-kube-api-access-mszzv" pod "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" (UID: "dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15"). InnerVolumeSpecName "kube-api-access-mszzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:47.195932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.195891 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.195932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.195926 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.195932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.195936 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.196211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.195945 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mszzv\" (UniqueName: \"kubernetes.io/projected/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15-kube-api-access-mszzv\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:47.229360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.229270 4924 generic.go:356] "Generic (PLEG): container finished" podID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerID="a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc" exitCode=0 Nov 03 13:39:47.229360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.229334 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" Nov 03 13:39:47.229539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.229362 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" event={"ID":"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15","Type":"ContainerDied","Data":"a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc"} Nov 03 13:39:47.229539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.229455 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt" event={"ID":"dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15","Type":"ContainerDied","Data":"e6ad493d182c17d8e9648bd4cedc49e06836a515a8d1360984af3717bd1ca6c4"} Nov 03 13:39:47.229539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.229488 4924 scope.go:122] "RemoveContainer" containerID="a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc" Nov 03 13:39:47.246205 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.246181 4924 scope.go:122] "RemoveContainer" containerID="a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc" Nov 03 13:39:47.246565 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:39:47.246546 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc\": container with ID starting with a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc not found: ID does not exist" containerID="a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc" Nov 03 13:39:47.246643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.246579 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc"} err="failed to get container status \"a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc\": rpc error: code = NotFound desc = could not find container \"a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc\": container with ID starting with a8009a29d057fd61dafcf4bd08738e57eb2e4270b40adbc0bf9d7bd8507a04bc not found: ID does not exist" Nov 03 13:39:47.398778 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.398743 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt"] Nov 03 13:39:47.408432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.408405 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-vkvnt"] Nov 03 13:39:47.519851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:47.519734 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:48.455096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.454559 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b"] Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455364 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455385 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455397 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455408 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455441 4924 cpu_manager.go:414] "containerMap: removing container" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455448 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455481 4924 cpu_manager.go:414] "containerMap: removing container" podUID="df36687a-57bd-4b91-8b5f-e8f095f700a9" containerName="maas-api" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455489 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df36687a-57bd-4b91-8b5f-e8f095f700a9" containerName="maas-api" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455505 4924 cpu_manager.go:414] "containerMap: removing container" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" Nov 03 13:39:48.455611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455512 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" Nov 03 13:39:48.456162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455647 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="337a993b-ad5a-4066-b7b0-417607091a40" containerName="istio-proxy" Nov 03 13:39:48.456162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455667 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="df36687a-57bd-4b91-8b5f-e8f095f700a9" containerName="maas-api" Nov 03 13:39:48.456162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455676 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" containerName="istio-proxy" Nov 03 13:39:48.456162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455689 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1bb9f30-a784-4180-aa25-64a3e8e79381" containerName="istio-proxy" Nov 03 13:39:48.456162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.455698 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17ef34d-0799-40fe-b18c-2681d8528771" containerName="istio-proxy" Nov 03 13:39:48.461023 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.460993 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.513931 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.513895 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-data\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.514115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.514051 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-certs\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.514217 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.514198 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-socket\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.514283 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.514268 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-envoy\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.514322 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.514305 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-token\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.514360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.514330 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1d972d-d193-4677-a9bb-9b259b8d0bca-istiod-ca-cert\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.514445 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.514417 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-credential-socket\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.514510 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.514495 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-podinfo\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.514564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.514547 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zks\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-kube-api-access-n2zks\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.526944 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.526903 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b"] Nov 03 13:39:48.576437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.576396 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6bd5c1-openshift-default-dockercfg-mbtns\"" Nov 03 13:39:48.578758 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.578700 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:48.616756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.616709 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-socket\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.616967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.616795 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-envoy\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.616967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.616833 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-token\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.616967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.616858 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1d972d-d193-4677-a9bb-9b259b8d0bca-istiod-ca-cert\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.616967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.616896 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-credential-socket\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.616967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.616926 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-podinfo\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.616967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.616953 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zks\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-kube-api-access-n2zks\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.617302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.617026 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-data\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.617302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.617130 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-certs\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.617682 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.617652 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-certs\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.618031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.617996 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-socket\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.618955 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:39:48.618922 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:39:48.619528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.619483 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-data\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.619674 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.619644 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1d972d-d193-4677-a9bb-9b259b8d0bca-istiod-ca-cert\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.620054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.620034 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-credential-socket\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.621305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.621270 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-envoy\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.622977 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.622875 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-podinfo\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.679570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.679538 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-token\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.679570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.679552 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zks\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-kube-api-access-n2zks\") pod \"e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.780477 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.780344 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:48.953182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:48.953138 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15" path="/var/lib/kubelet/pods/dcb36b5d-c3cc-43f6-8ce9-1ead0fb21c15/volumes" Nov 03 13:39:49.082800 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:49.082719 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:49.219023 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:49.218982 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b"] Nov 03 13:39:49.229226 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:49.228511 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:49.229226 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:49.228603 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:49.229226 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:49.228646 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:49.251949 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:49.251894 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:49.253715 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:49.253684 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" event={"ID":"be1d972d-d193-4677-a9bb-9b259b8d0bca","Type":"ContainerStarted","Data":"7e195ff6580fcc0abb74181768f72c4ace49fc14a43fb43859177037d446cd6f"} Nov 03 13:39:50.262996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:50.262954 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" event={"ID":"be1d972d-d193-4677-a9bb-9b259b8d0bca","Type":"ContainerStarted","Data":"697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17"} Nov 03 13:39:50.426530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:50.426454 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" podStartSLOduration=2.426436717 podStartE2EDuration="2.426436717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:50.425507734 +0000 UTC m=+2123.619911327" watchObservedRunningTime="2026-11-03 13:39:50.426436717 +0000 UTC m=+2123.620840319" Nov 03 13:39:50.781545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:50.781509 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:50.787334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:50.787309 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:51.272714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:51.272668 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:51.273849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:51.273818 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:39:53.253799 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:53.253769 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:53.478291 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:53.478257 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:53.632943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:53.632813 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:54.349792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:54.349760 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:56.268809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:56.268765 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:57.424896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:57.424860 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:57.573587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:57.573535 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:57.722298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:57.722215 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:58.565640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.565603 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7"] Nov 03 13:39:58.570867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.570841 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633212 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-data\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633311 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-istio-token\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633373 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-socket\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633411 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de8c30-2906-4737-8dec-e81717588084-istio-podinfo\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633467 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de8c30-2906-4737-8dec-e81717588084-istiod-ca-cert\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633491 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-certs\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633518 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-credential-socket\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633557 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-envoy\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.633625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.633613 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsjrg\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-kube-api-access-bsjrg\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.635377 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.635343 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7"] Nov 03 13:39:58.643282 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.643212 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:58.676215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.676185 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-007b0d-openshift-default-dockercfg-vxqlw\"" Nov 03 13:39:58.735088 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735040 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-istio-token\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.735299 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735151 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-socket\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.735299 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735186 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de8c30-2906-4737-8dec-e81717588084-istio-podinfo\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.735299 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735230 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de8c30-2906-4737-8dec-e81717588084-istiod-ca-cert\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.735299 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735249 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-certs\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.735299 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735271 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-credential-socket\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.735299 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735299 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-envoy\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.735856 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735334 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsjrg\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-kube-api-access-bsjrg\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.735856 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735395 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-data\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.736059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.735897 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-data\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.736464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.736442 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-socket\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.737038 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.737010 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-credential-socket\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.737206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.737152 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-certs\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.738047 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.738025 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de8c30-2906-4737-8dec-e81717588084-istiod-ca-cert\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.739301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.739274 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-envoy\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.739460 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.739439 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de8c30-2906-4737-8dec-e81717588084-istio-podinfo\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.790272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.790242 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-istio-token\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.790787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.790757 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsjrg\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-kube-api-access-bsjrg\") pod \"e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:58.803128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.801631 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:39:58.886187 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:58.886093 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:39:59.312118 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:59.312083 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7"] Nov 03 13:39:59.312822 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:39:59.312786 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52de8c30_2906_4737_8dec_e81717588084.slice/crio-fc08ee6ca2abb746f639a1fae60516e43d6af3eb368cc8700c0c9ea69e70c8ca WatchSource:0}: Error finding container fc08ee6ca2abb746f639a1fae60516e43d6af3eb368cc8700c0c9ea69e70c8ca: Status 404 returned error can't find the container with id fc08ee6ca2abb746f639a1fae60516e43d6af3eb368cc8700c0c9ea69e70c8ca Nov 03 13:39:59.315331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:59.315300 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:59.315407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:59.315364 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:59.315407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:59.315394 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:39:59.333995 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:39:59.333963 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" event={"ID":"52de8c30-2906-4737-8dec-e81717588084","Type":"ContainerStarted","Data":"fc08ee6ca2abb746f639a1fae60516e43d6af3eb368cc8700c0c9ea69e70c8ca"} Nov 03 13:40:00.175659 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:00.175621 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:00.343315 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:00.343276 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" event={"ID":"52de8c30-2906-4737-8dec-e81717588084","Type":"ContainerStarted","Data":"bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524"} Nov 03 13:40:00.506454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:00.506347 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" podStartSLOduration=2.506332201 podStartE2EDuration="2.506332201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:00.504921888 +0000 UTC m=+2133.699325481" watchObservedRunningTime="2026-11-03 13:40:00.506332201 +0000 UTC m=+2133.700735795" Nov 03 13:40:00.887014 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:00.886932 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:40:00.892324 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:00.892297 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:40:01.350002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:01.349968 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:40:01.350934 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:01.350915 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:40:03.332295 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:03.332214 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:03.547961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:03.547900 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:03.721557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:03.721523 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:05.926522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:05.926470 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:07.239576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:07.239540 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:07.417797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:07.417749 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:13.200816 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:13.200783 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:18.559929 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:40:18.559895 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:40:18.653227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:18.653190 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:23.516337 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.516304 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq"] Nov 03 13:40:23.520405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.520385 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.584304 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.584273 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8d8af2-serving-cert\"" Nov 03 13:40:23.588502 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.588470 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq"] Nov 03 13:40:23.631414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.631380 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-metrics-certs\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.631602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.631428 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.631602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.631515 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hr28\" (UniqueName: \"kubernetes.io/projected/93637a67-e509-46c1-8951-eae64693d920-kube-api-access-7hr28\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.631602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.631592 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93637a67-e509-46c1-8951-eae64693d920-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.681407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.681369 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:23.733206 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.733173 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-metrics-certs\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.733407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.733219 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.733407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.733277 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hr28\" (UniqueName: \"kubernetes.io/projected/93637a67-e509-46c1-8951-eae64693d920-kube-api-access-7hr28\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.733407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.733341 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93637a67-e509-46c1-8951-eae64693d920-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.734223 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.734188 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93637a67-e509-46c1-8951-eae64693d920-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.735809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.735784 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-metrics-certs\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.736001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.735976 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.790358 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.790274 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hr28\" (UniqueName: \"kubernetes.io/projected/93637a67-e509-46c1-8951-eae64693d920-kube-api-access-7hr28\") pod \"maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.834713 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.834679 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:23.836412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.836382 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:23.966333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:23.966300 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:24.208060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:24.208033 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq"] Nov 03 13:40:24.209280 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:40:24.209253 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93637a67_e509_46c1_8951_eae64693d920.slice/crio-738ff28fb631941e991b1e3f11b552f896b2cee9d13e17867f2d11a58250e388 WatchSource:0}: Error finding container 738ff28fb631941e991b1e3f11b552f896b2cee9d13e17867f2d11a58250e388: Status 404 returned error can't find the container with id 738ff28fb631941e991b1e3f11b552f896b2cee9d13e17867f2d11a58250e388 Nov 03 13:40:24.533044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:24.532960 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" event={"ID":"93637a67-e509-46c1-8951-eae64693d920","Type":"ContainerStarted","Data":"738ff28fb631941e991b1e3f11b552f896b2cee9d13e17867f2d11a58250e388"} Nov 03 13:40:25.542378 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:25.542290 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" event={"ID":"93637a67-e509-46c1-8951-eae64693d920","Type":"ContainerStarted","Data":"6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3"} Nov 03 13:40:25.542854 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:25.542563 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:25.660634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:25.660569 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" podStartSLOduration=1.610362684 podStartE2EDuration="2.660549481s" totalImagesPullingTime="1.050186797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:25.659687266 +0000 UTC m=+2158.854090860" watchObservedRunningTime="2026-11-03 13:40:25.660549481 +0000 UTC m=+2158.854953079" Nov 03 13:40:28.714339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:28.714289 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:28.986514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:28.986426 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:31.555443 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:31.555411 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:40:34.122643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:34.122594 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:34.291594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:34.291555 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:34.413714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:34.413670 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:39.578366 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:39.578330 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:46.499378 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:46.499318 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:48.560973 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:40:48.560923 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:40:51.046981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:51.046942 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:51.580404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:51.580363 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:51.710048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:51.710012 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:52.609218 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:52.609185 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:52.748951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:52.748913 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:52.980116 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:52.980086 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:54.249085 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:54.249045 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:54.750392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:54.750318 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:56.275671 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:56.275634 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:57.143597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:57.143563 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:57.329132 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:57.328214 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:58.579017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:58.578984 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:58.745743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:58.745710 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:40:58.862575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:40:58.862472 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:01.453503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:01.453464 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:01.633430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:01.633396 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:01.793172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:01.793081 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:04.543531 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:04.543489 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:04.784141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:04.784107 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:04.958435 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:04.958379 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:10.270939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:10.270855 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:14.313494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:14.313456 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:15.345298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.345258 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq"] Nov 03 13:41:15.351463 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.351436 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.401511 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.401481 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-007b0d-serving-cert\"" Nov 03 13:41:15.404323 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.404296 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq"] Nov 03 13:41:15.477969 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.477931 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30158b4-a3dc-430b-820e-50d7f0160feb-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.478376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.478350 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkts9\" (UniqueName: \"kubernetes.io/projected/b30158b4-a3dc-430b-820e-50d7f0160feb-kube-api-access-mkts9\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.478568 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.478550 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-metrics-certs\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.478696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.478670 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.530656 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.530620 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:15.580034 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.579995 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.580241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.580154 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30158b4-a3dc-430b-820e-50d7f0160feb-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.580241 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:41:15.580175 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-serving-cert: secret "maas-api-e2e-praxis-b-007b0d-serving-cert" not found Nov 03 13:41:15.580241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.580206 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkts9\" (UniqueName: \"kubernetes.io/projected/b30158b4-a3dc-430b-820e-50d7f0160feb-kube-api-access-mkts9\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.580241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.580238 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-metrics-certs\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.580400 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:41:15.580256 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls podName:b30158b4-a3dc-430b-820e-50d7f0160feb nodeName:}" failed. No retries permitted until 2026-11-03 13:41:16.080235943 +0000 UTC m=+2209.274639514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls") pod "maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" (UID: "b30158b4-a3dc-430b-820e-50d7f0160feb") : secret "maas-api-e2e-praxis-b-007b0d-serving-cert" not found Nov 03 13:41:15.580859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.580842 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30158b4-a3dc-430b-820e-50d7f0160feb-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.582686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.582666 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-metrics-certs\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.637100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.637012 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkts9\" (UniqueName: \"kubernetes.io/projected/b30158b4-a3dc-430b-820e-50d7f0160feb-kube-api-access-mkts9\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:15.696779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.696745 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:15.823340 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:15.823294 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:16.085660 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:16.085623 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:16.088131 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:16.088105 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls\") pod \"maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:16.269134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:16.269093 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:16.636224 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:16.636193 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq"] Nov 03 13:41:16.637841 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:41:16.637810 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30158b4_a3dc_430b_820e_50d7f0160feb.slice/crio-0cfc9275889b1d6f5570a0f64e2c905adf3c5291b20bada5a8ee9e120bf47b5f WatchSource:0}: Error finding container 0cfc9275889b1d6f5570a0f64e2c905adf3c5291b20bada5a8ee9e120bf47b5f: Status 404 returned error can't find the container with id 0cfc9275889b1d6f5570a0f64e2c905adf3c5291b20bada5a8ee9e120bf47b5f Nov 03 13:41:16.938669 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:16.938632 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" event={"ID":"b30158b4-a3dc-430b-820e-50d7f0160feb","Type":"ContainerStarted","Data":"0cfc9275889b1d6f5570a0f64e2c905adf3c5291b20bada5a8ee9e120bf47b5f"} Nov 03 13:41:17.946830 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:17.946790 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" event={"ID":"b30158b4-a3dc-430b-820e-50d7f0160feb","Type":"ContainerStarted","Data":"eb32d7cbc1f6c3023d0cf0bb6c90ec1efbd16c4f26f4fb7f3966cfc21d25f5d8"} Nov 03 13:41:17.947263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:17.946857 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:18.067308 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:18.067242 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" podStartSLOduration=2.134947796 podStartE2EDuration="3.06722323s" totalImagesPullingTime="932.275434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:18.064732644 +0000 UTC m=+2211.259136237" watchObservedRunningTime="2026-11-03 13:41:18.06722323 +0000 UTC m=+2211.261626826" Nov 03 13:41:18.560765 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:41:18.560726 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:41:20.991402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:20.991364 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:23.960871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:23.960834 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:41:27.808510 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:27.808476 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:29.040701 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:29.040655 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:32.897718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:32.897680 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:38.048096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:38.047284 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:42.162359 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:42.162316 4924 generic.go:356] "Generic (PLEG): container finished" podID="52de8c30-2906-4737-8dec-e81717588084" containerID="bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524" exitCode=0 Nov 03 13:41:42.163783 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:42.162390 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" event={"ID":"52de8c30-2906-4737-8dec-e81717588084","Type":"ContainerDied","Data":"bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524"} Nov 03 13:41:42.163783 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:42.162840 4924 scope.go:122] "RemoveContainer" containerID="bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524" Nov 03 13:41:42.164590 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:42.164552 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:41:42.164686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:42.164668 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:41:42.164747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:42.164719 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:41:43.173984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.173943 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" event={"ID":"52de8c30-2906-4737-8dec-e81717588084","Type":"ContainerStarted","Data":"66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5"} Nov 03 13:41:43.176219 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.176193 4924 generic.go:356] "Generic (PLEG): container finished" podID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerID="697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17" exitCode=0 Nov 03 13:41:43.176481 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.176273 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" event={"ID":"be1d972d-d193-4677-a9bb-9b259b8d0bca","Type":"ContainerDied","Data":"697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17"} Nov 03 13:41:43.176744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.176727 4924 scope.go:122] "RemoveContainer" containerID="697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17" Nov 03 13:41:43.178522 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.178487 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:41:43.178643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.178555 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:41:43.178643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.178586 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:41:43.188738 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.188713 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:43.886383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.886346 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:41:43.887734 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.887694 4924 patch_prober.go:29] interesting pod/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 13:41:43.887898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:43.887787 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 13:41:44.191741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:44.191705 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" event={"ID":"be1d972d-d193-4677-a9bb-9b259b8d0bca","Type":"ContainerStarted","Data":"c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e"} Nov 03 13:41:44.781367 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:44.781281 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:41:44.781969 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:44.781657 4924 patch_prober.go:29] interesting pod/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:41:44.781969 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:44.781709 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:41:44.888122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:44.887998 4924 patch_prober.go:29] interesting pod/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 13:41:44.888122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:44.888046 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 13:41:45.782093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:45.781585 4924 patch_prober.go:29] interesting pod/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:41:45.782093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:45.781643 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:41:45.886802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:45.886754 4924 patch_prober.go:29] interesting pod/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 13:41:45.886980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:45.886824 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 13:41:46.352647 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:46.352413 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:41:46.781672 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:46.781623 4924 patch_prober.go:29] interesting pod/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:41:46.781866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:46.781695 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:41:46.886909 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:46.886877 4924 patch_prober.go:29] interesting pod/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 13:41:46.887388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:46.887173 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 13:41:47.783694 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:47.783654 4924 patch_prober.go:29] interesting pod/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:41:47.784020 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:47.783976 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:41:47.886881 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:47.886845 4924 patch_prober.go:29] interesting pod/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 13:41:47.887126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:47.886897 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 13:41:48.600729 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:41:48.600662 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:41:48.780748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:48.780701 4924 patch_prober.go:29] interesting pod/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:41:48.780952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:48.780763 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:41:48.891189 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:48.891111 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:41:48.892191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:48.892168 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:41:48.905305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:48.905276 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:49.046095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:49.044622 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:49.785980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:49.785952 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:41:49.786342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:49.786323 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:41:49.786995 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:49.786975 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:41:50.191799 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:50.191765 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:50.803093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:50.802252 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:51.000204 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:51.000157 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:54.492837 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:54.492800 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:54.837309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:54.837220 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:55.018186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:55.018148 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:55.166298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:55.166263 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:59.468885 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:59.468826 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:41:59.943864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:41:59.943826 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:00.124478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:00.124434 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:01.536026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:01.535948 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:01.688007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:01.687972 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:02.165891 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:02.165839 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:07.495013 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.494977 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:07.644798 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.644760 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5"] Nov 03 13:42:07.647938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.647917 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.697027 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.696993 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a770b9\"/\"openshift-service-ca.crt\"" Nov 03 13:42:07.720564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.720529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:07.743791 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.743657 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a770b9\"/\"default-dockercfg-jgs5j\"" Nov 03 13:42:07.743791 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.743747 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a770b9\"/\"kube-root-ca.crt\"" Nov 03 13:42:07.743791 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.743767 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a770b9\"/\"ipp-route-a770b9-kserve-self-signed-certs\"" Nov 03 13:42:07.749139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.749029 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5"] Nov 03 13:42:07.764551 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.764518 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-model-cache\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.764716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.764560 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-home\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.764716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.764598 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26603eeb-8c51-4378-bdeb-3774e2455395-tls-certs\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.764812 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.764731 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-tmp-dir\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.764812 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.764783 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-dshm\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.764903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.764882 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w8g6\" (UniqueName: \"kubernetes.io/projected/26603eeb-8c51-4378-bdeb-3774e2455395-kube-api-access-9w8g6\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.866389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.866344 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-tmp-dir\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.866601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.866445 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-dshm\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.866601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.866520 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8g6\" (UniqueName: \"kubernetes.io/projected/26603eeb-8c51-4378-bdeb-3774e2455395-kube-api-access-9w8g6\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.866718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.866601 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-model-cache\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.866718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.866639 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-home\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.866718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.866703 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26603eeb-8c51-4378-bdeb-3774e2455395-tls-certs\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.866870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.866820 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-tmp-dir\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.867134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.867111 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-model-cache\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.867394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.867372 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-home\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.869664 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.869634 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-dshm\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.869935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.869910 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26603eeb-8c51-4378-bdeb-3774e2455395-tls-certs\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.921581 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.921550 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8g6\" (UniqueName: \"kubernetes.io/projected/26603eeb-8c51-4378-bdeb-3774e2455395-kube-api-access-9w8g6\") pod \"ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:07.959861 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:07.959826 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:08.341610 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:08.341572 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5"] Nov 03 13:42:08.342434 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:42:08.342396 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26603eeb_8c51_4378_bdeb_3774e2455395.slice/crio-fa9299ea17543892d425cb645f8df160529e4909f0ba7071385793994374ae05 WatchSource:0}: Error finding container fa9299ea17543892d425cb645f8df160529e4909f0ba7071385793994374ae05: Status 404 returned error can't find the container with id fa9299ea17543892d425cb645f8df160529e4909f0ba7071385793994374ae05 Nov 03 13:42:08.370163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:08.370121 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" event={"ID":"26603eeb-8c51-4378-bdeb-3774e2455395","Type":"ContainerStarted","Data":"fa9299ea17543892d425cb645f8df160529e4909f0ba7071385793994374ae05"} Nov 03 13:42:09.378947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:09.378909 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" event={"ID":"26603eeb-8c51-4378-bdeb-3774e2455395","Type":"ContainerStarted","Data":"21f33220c572d1a45277306fd05afe580058be3d51adfe99302cceb27c027e2d"} Nov 03 13:42:09.489953 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:09.489901 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" podStartSLOduration=2.489887006 podStartE2EDuration="2.489887006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:09.488121356 +0000 UTC m=+2262.682524950" watchObservedRunningTime="2026-11-03 13:42:09.489887006 +0000 UTC m=+2262.684290599" Nov 03 13:42:09.960343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:09.960303 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:12.486843 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:12.486793 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:17.529410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:17.529355 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:17.672957 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:17.672881 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:17.790416 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:17.790321 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:17.960777 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:17.960744 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:17.975428 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:17.975383 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:18.453030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:18.452991 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:42:18.559245 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:42:18.559203 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:42:22.125240 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:22.125202 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:23.243416 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:23.243379 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:23.414431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:23.414396 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:24.479888 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:24.479847 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:26.878191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:26.878152 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:27.032330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:27.032296 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:28.482534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:28.482488 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:29.273024 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:29.272988 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:29.407530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:29.407496 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:34.009492 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:34.009456 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8"] Nov 03 13:42:34.009981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:34.009651 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" containerID="cri-o://96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e" gracePeriod=30 Nov 03 13:42:34.098947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:34.098912 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:37.188607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:37.188520 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:42:37.188607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:37.188582 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:42:39.278804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.278737 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:42:39.392178 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392142 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-data\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392367 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392184 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-credential-socket\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392367 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392240 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fae73af-85a0-4129-909d-6be7992d360d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fae73af-85a0-4129-909d-6be7992d360d-istio-podinfo\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392367 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392308 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-istio-token\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-istio-token\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392367 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392361 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-kube-api-access-2drgz\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-kube-api-access-2drgz\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392406 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-socket\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392448 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-certs\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392461 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-data" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:39.392598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392529 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-envoy\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392589 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fae73af-85a0-4129-909d-6be7992d360d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fae73af-85a0-4129-909d-6be7992d360d-istiod-ca-cert\") pod \"2fae73af-85a0-4129-909d-6be7992d360d\" (UID: \"2fae73af-85a0-4129-909d-6be7992d360d\") " Nov 03 13:42:39.392875 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392731 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-socket" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:39.392875 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.392752 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-certs" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:39.393054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.393027 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fae73af-85a0-4129-909d-6be7992d360d-istiod-ca-cert" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:39.393169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.393142 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-credential-socket" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:39.393262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.393225 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.393262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.393246 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.393262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.393255 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fae73af-85a0-4129-909d-6be7992d360d-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.393426 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.393265 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.394821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.394796 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fae73af-85a0-4129-909d-6be7992d360d-istio-podinfo" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:42:39.394988 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.394971 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-istio-token" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:39.395116 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.395064 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-kube-api-access-2drgz" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "kube-api-access-2drgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:39.395116 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.395106 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-envoy" pod "2fae73af-85a0-4129-909d-6be7992d360d" (UID: "2fae73af-85a0-4129-909d-6be7992d360d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:39.493923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.493886 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.493923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.493922 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fae73af-85a0-4129-909d-6be7992d360d-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.493923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.493932 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.494194 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.493943 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2drgz\" (UniqueName: \"kubernetes.io/projected/2fae73af-85a0-4129-909d-6be7992d360d-kube-api-access-2drgz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.494194 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.493954 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fae73af-85a0-4129-909d-6be7992d360d-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:39.584757 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.584665 4924 generic.go:356] "Generic (PLEG): container finished" podID="2fae73af-85a0-4129-909d-6be7992d360d" containerID="96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e" exitCode=0 Nov 03 13:42:39.584757 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.584723 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" Nov 03 13:42:39.585032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.584752 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" event={"ID":"2fae73af-85a0-4129-909d-6be7992d360d","Type":"ContainerDied","Data":"96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e"} Nov 03 13:42:39.585032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.584822 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8" event={"ID":"2fae73af-85a0-4129-909d-6be7992d360d","Type":"ContainerDied","Data":"fb2f548da46cc8940cdb4407125316aae192f1ee00fc76c221427ab76465d4ad"} Nov 03 13:42:39.585032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.584854 4924 scope.go:122] "RemoveContainer" containerID="96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e" Nov 03 13:42:39.596545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.596507 4924 scope.go:122] "RemoveContainer" containerID="96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e" Nov 03 13:42:39.597854 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:42:39.596933 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e\": container with ID starting with 96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e not found: ID does not exist" containerID="96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e" Nov 03 13:42:39.597854 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.596971 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e"} err="failed to get container status \"96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e\": rpc error: code = NotFound desc = could not find container \"96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e\": container with ID starting with 96a6c678e1ccf974fc986aef02e90172d7b634417708cba7f29559499873ad2e not found: ID does not exist" Nov 03 13:42:39.697795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.697766 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:39.742195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.742162 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8"] Nov 03 13:42:39.747612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:39.747581 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-j4zl8"] Nov 03 13:42:40.963064 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:40.963028 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fae73af-85a0-4129-909d-6be7992d360d" path="/var/lib/kubelet/pods/2fae73af-85a0-4129-909d-6be7992d360d/volumes" Nov 03 13:42:41.902830 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:41.902793 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:44.003152 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:44.003100 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:45.849782 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:45.849743 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:45.972281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:45.972247 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:47.202786 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:47.202748 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:48.560894 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:42:48.560858 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:42:49.868988 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:49.868954 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:51.236849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:51.236801 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:51.363087 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:51.363037 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:52.707054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:52.707021 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:55.623913 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:55.623880 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:57.627441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:57.627400 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:42:58.402030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:42:58.401967 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:04.063895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:04.063862 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:09.575339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:09.575291 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:13.249923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:13.249891 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:13.794435 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:13.794402 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:16.364305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.364268 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf"] Nov 03 13:43:16.364924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.364904 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" Nov 03 13:43:16.364924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.364924 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" Nov 03 13:43:16.365115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.365059 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fae73af-85a0-4129-909d-6be7992d360d" containerName="istio-proxy" Nov 03 13:43:16.368634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.368612 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.418676 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.418646 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-84e22c-serving-cert\"" Nov 03 13:43:16.435987 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.435945 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.436278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.436249 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pg6\" (UniqueName: \"kubernetes.io/projected/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-kube-api-access-49pg6\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.436352 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.436318 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.436406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.436363 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-metrics-certs\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.469408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.469372 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf"] Nov 03 13:43:16.537537 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.537495 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-metrics-certs\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.537726 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.537631 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.537797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.537742 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pg6\" (UniqueName: \"kubernetes.io/projected/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-kube-api-access-49pg6\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.537858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.537797 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.537939 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:43:16.537918 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-serving-cert: secret "maas-api-e2e-praxis-b-84e22c-serving-cert" not found Nov 03 13:43:16.538007 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:43:16.537998 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls podName:d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8 nodeName:}" failed. No retries permitted until 2026-11-03 13:43:17.037976285 +0000 UTC m=+2330.232379877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls") pod "maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" (UID: "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8") : secret "maas-api-e2e-praxis-b-84e22c-serving-cert" not found Nov 03 13:43:16.538323 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.538303 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.539852 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.539830 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-metrics-certs\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.569770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.569736 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:16.595527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.595493 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pg6\" (UniqueName: \"kubernetes.io/projected/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-kube-api-access-49pg6\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:16.727285 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.727250 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:16.868096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:16.867370 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:17.043514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:17.043422 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:17.046618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:17.046585 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:17.283864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:17.283824 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:17.661234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:17.661194 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf"] Nov 03 13:43:17.662292 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:43:17.662264 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a13f2a_9835_46b2_bbf5_52c7c2bee2f8.slice/crio-206d53e091518f3346b0e890d9701da196f49f2591735481eb41eaf9563ce69f WatchSource:0}: Error finding container 206d53e091518f3346b0e890d9701da196f49f2591735481eb41eaf9563ce69f: Status 404 returned error can't find the container with id 206d53e091518f3346b0e890d9701da196f49f2591735481eb41eaf9563ce69f Nov 03 13:43:17.663841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:17.663821 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:43:17.831138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:17.831095 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" event={"ID":"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8","Type":"ContainerStarted","Data":"206d53e091518f3346b0e890d9701da196f49f2591735481eb41eaf9563ce69f"} Nov 03 13:43:18.458555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:18.458510 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:18.571147 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:43:18.571105 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:43:18.584646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:18.584615 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:18.841997 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:18.841898 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" event={"ID":"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8","Type":"ContainerStarted","Data":"ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8"} Nov 03 13:43:18.842508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:18.842029 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:18.954020 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:18.953965 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" podStartSLOduration=2.058660736 podStartE2EDuration="2.95394985s" totalImagesPullingTime="895.289114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:18.952378005 +0000 UTC m=+2332.146781600" watchObservedRunningTime="2026-11-03 13:43:18.95394985 +0000 UTC m=+2332.148353478" Nov 03 13:43:21.278536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:21.278497 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:21.798234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:21.798201 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:21.941852 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:21.941814 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:23.119982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:23.119946 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:24.855003 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:24.854970 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:43:27.823527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:27.823494 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:29.743932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:29.743897 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:29.879850 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:29.879817 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:30.822750 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:30.822714 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:31.027319 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:31.027282 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:32.593662 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:32.593577 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:33.282164 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:33.282118 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2"] Nov 03 13:43:33.282391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:33.282363 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" containerID="cri-o://8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1" gracePeriod=30 Nov 03 13:43:33.329693 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:33.329657 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:34.001316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:34.001252 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:34.421728 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:34.421693 4924 patch_prober.go:29] interesting pod/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:43:34.421910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:34.421782 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:43:34.492743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:34.492709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:37.706739 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:37.706700 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:38.564736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.564666 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:43:38.707305 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.707271 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:38.728784 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.728748 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-socket\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.728966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.728814 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-kube-api-access-n8m9r\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-kube-api-access-n8m9r\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.728966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.728907 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77521ec-beb3-4383-b2e2-e2bb66557174-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77521ec-beb3-4383-b2e2-e2bb66557174-istiod-ca-cert\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.728966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.728941 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-podinfo\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.729173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.728991 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-credential-socket\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.729173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.729037 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-data\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.729173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.729116 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-envoy\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.729325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.729181 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-certs\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.729325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.729199 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-token\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-token\") pod \"d77521ec-beb3-4383-b2e2-e2bb66557174\" (UID: \"d77521ec-beb3-4383-b2e2-e2bb66557174\") " Nov 03 13:43:38.730330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.729627 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-credential-socket" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:38.730330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.729819 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.730330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.729989 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77521ec-beb3-4383-b2e2-e2bb66557174-istiod-ca-cert" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:38.730330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.730229 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-socket" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:38.730330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.730286 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-data" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:38.730721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.730428 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-certs" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:38.732446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.732420 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-kube-api-access-n8m9r" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "kube-api-access-n8m9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:38.738501 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.738465 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-podinfo" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:43:38.738610 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.738564 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-envoy" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:38.738659 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.738623 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-token" pod "d77521ec-beb3-4383-b2e2-e2bb66557174" (UID: "d77521ec-beb3-4383-b2e2-e2bb66557174"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:38.831372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.831276 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.831372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.831308 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8m9r\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-kube-api-access-n8m9r\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.831372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.831324 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77521ec-beb3-4383-b2e2-e2bb66557174-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.831372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.831336 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.831372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.831349 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.831372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.831361 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.831372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.831375 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77521ec-beb3-4383-b2e2-e2bb66557174-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.831716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.831387 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77521ec-beb3-4383-b2e2-e2bb66557174-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:38.938432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:38.938392 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:39.035346 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.035307 4924 generic.go:356] "Generic (PLEG): container finished" podID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerID="8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1" exitCode=0 Nov 03 13:43:39.035607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.035445 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" Nov 03 13:43:39.035932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.035467 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" event={"ID":"d77521ec-beb3-4383-b2e2-e2bb66557174","Type":"ContainerDied","Data":"8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1"} Nov 03 13:43:39.036036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.035938 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2" event={"ID":"d77521ec-beb3-4383-b2e2-e2bb66557174","Type":"ContainerDied","Data":"9b247fd34eccd260c84bdb67a9938703127889a86b01228fb7b60fbe26bca58f"} Nov 03 13:43:39.036036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.035983 4924 scope.go:122] "RemoveContainer" containerID="8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1" Nov 03 13:43:39.046286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.046262 4924 scope.go:122] "RemoveContainer" containerID="09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2" Nov 03 13:43:39.055924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.055899 4924 scope.go:122] "RemoveContainer" containerID="8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1" Nov 03 13:43:39.056288 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:43:39.056262 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1\": container with ID starting with 8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1 not found: ID does not exist" containerID="8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1" Nov 03 13:43:39.056386 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.056301 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1"} err="failed to get container status \"8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1\": rpc error: code = NotFound desc = could not find container \"8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1\": container with ID starting with 8337085ac7a6aa7ff4ab519761e1e6f6804e07b3f83f529283ba8590482f27e1 not found: ID does not exist" Nov 03 13:43:39.056386 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.056329 4924 scope.go:122] "RemoveContainer" containerID="09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2" Nov 03 13:43:39.056612 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:43:39.056594 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2\": container with ID starting with 09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2 not found: ID does not exist" containerID="09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2" Nov 03 13:43:39.056689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.056621 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2"} err="failed to get container status \"09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2\": rpc error: code = NotFound desc = could not find container \"09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2\": container with ID starting with 09aa1f35c2ae9d4f88d1527c386666a0daa8a73e5a27a3685ae8eddb7c297fa2 not found: ID does not exist" Nov 03 13:43:39.201083 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.201044 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2"] Nov 03 13:43:39.247991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.247955 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-02f3a0-openshift-default-5489d8ddd7-b28d2"] Nov 03 13:43:39.282241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:39.282206 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:40.309119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:40.309063 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:40.858562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:40.858523 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:40.879061 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:40.879030 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w"] Nov 03 13:43:40.879347 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:40.879309 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" containerID="cri-o://39a2119f19c0ceeab811cb9218f4e1d92805832f5995170b05eb7fca5ad46746" gracePeriod=30 Nov 03 13:43:41.011314 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:41.011277 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" path="/var/lib/kubelet/pods/d77521ec-beb3-4383-b2e2-e2bb66557174/volumes" Nov 03 13:43:41.591778 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:41.591747 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:43.651388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:43.651352 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:44.717908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:44.717871 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:43:44.718302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:44.717927 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:43:45.532245 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:45.532209 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:46.117961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.117633 4924 generic.go:356] "Generic (PLEG): container finished" podID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerID="39a2119f19c0ceeab811cb9218f4e1d92805832f5995170b05eb7fca5ad46746" exitCode=0 Nov 03 13:43:46.117961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.117828 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" event={"ID":"b16c9379-dc86-4ecd-a669-ebe078128f6f","Type":"ContainerDied","Data":"39a2119f19c0ceeab811cb9218f4e1d92805832f5995170b05eb7fca5ad46746"} Nov 03 13:43:46.117961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.117874 4924 scope.go:122] "RemoveContainer" containerID="a93737a996f930070e509bf077d61c4f764e93e204ef53a23fade321c2b1a3b1" Nov 03 13:43:46.152200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.152180 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:43:46.239736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.239699 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-podinfo\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.239900 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.239775 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16c9379-dc86-4ecd-a669-ebe078128f6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16c9379-dc86-4ecd-a669-ebe078128f6f-istiod-ca-cert\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.239900 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.239800 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-token\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-token\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.239985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.239957 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-kube-api-access-4cg6l\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-kube-api-access-4cg6l\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.240056 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240039 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-envoy\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.240195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240176 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-data\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.240284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240226 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-certs\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.240284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240224 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16c9379-dc86-4ecd-a669-ebe078128f6f-istiod-ca-cert" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:46.240284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240269 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-credential-socket\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.240469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240343 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-socket\") pod \"b16c9379-dc86-4ecd-a669-ebe078128f6f\" (UID: \"b16c9379-dc86-4ecd-a669-ebe078128f6f\") " Nov 03 13:43:46.240469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240426 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-data" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:46.240575 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240513 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-certs" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:46.240711 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240689 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-credential-socket" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:46.240778 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240707 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-socket" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:46.240866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240842 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b16c9379-dc86-4ecd-a669-ebe078128f6f-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.240930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240873 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.240930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240888 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.240930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240904 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.240930 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.240918 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.242179 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.242159 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-token" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:46.242271 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.242254 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-envoy" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:46.242490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.242460 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-podinfo" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:43:46.242490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.242464 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-kube-api-access-4cg6l" pod "b16c9379-dc86-4ecd-a669-ebe078128f6f" (UID: "b16c9379-dc86-4ecd-a669-ebe078128f6f"). InnerVolumeSpecName "kube-api-access-4cg6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:46.342051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.341888 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.342051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.341927 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.342051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.341943 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cg6l\" (UniqueName: \"kubernetes.io/projected/b16c9379-dc86-4ecd-a669-ebe078128f6f-kube-api-access-4cg6l\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.342051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.341959 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b16c9379-dc86-4ecd-a669-ebe078128f6f-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:46.696508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:46.696471 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:47.125482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:47.125449 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" Nov 03 13:43:47.125965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:47.125477 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w" event={"ID":"b16c9379-dc86-4ecd-a669-ebe078128f6f","Type":"ContainerDied","Data":"bd1fe86647264e2b09b3bf623b382e2182d43b84821c35b51062ae0d67f0aba5"} Nov 03 13:43:47.125965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:47.125547 4924 scope.go:122] "RemoveContainer" containerID="39a2119f19c0ceeab811cb9218f4e1d92805832f5995170b05eb7fca5ad46746" Nov 03 13:43:47.439762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:47.429640 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:47.466132 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:47.463546 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w"] Nov 03 13:43:47.470276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:47.468145 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-21f111-openshift-default-db87bcfc-z4h8w"] Nov 03 13:43:47.666437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:47.666402 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:47.980382 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:47.980349 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:48.558369 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:43:48.558321 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:43:48.951148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:48.951118 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" path="/var/lib/kubelet/pods/b16c9379-dc86-4ecd-a669-ebe078128f6f/volumes" Nov 03 13:43:49.536104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:49.529873 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:49.704209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:49.704175 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:49.947392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:49.947346 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:52.269060 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:52.269013 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:52.537994 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:52.537900 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:52.726397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:52.726359 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:54.778186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:54.778151 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:54.824993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:54.824959 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq"] Nov 03 13:43:54.825214 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:54.825190 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" podUID="b30158b4-a3dc-430b-820e-50d7f0160feb" containerName="maas-api" containerID="cri-o://eb32d7cbc1f6c3023d0cf0bb6c90ec1efbd16c4f26f4fb7f3966cfc21d25f5d8" gracePeriod=30 Nov 03 13:43:54.944017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:54.943986 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:55.149273 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.149179 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:55.202158 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.192707 4924 generic.go:356] "Generic (PLEG): container finished" podID="b30158b4-a3dc-430b-820e-50d7f0160feb" containerID="eb32d7cbc1f6c3023d0cf0bb6c90ec1efbd16c4f26f4fb7f3966cfc21d25f5d8" exitCode=0 Nov 03 13:43:55.202158 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.192974 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" event={"ID":"b30158b4-a3dc-430b-820e-50d7f0160feb","Type":"ContainerDied","Data":"eb32d7cbc1f6c3023d0cf0bb6c90ec1efbd16c4f26f4fb7f3966cfc21d25f5d8"} Nov 03 13:43:55.202158 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.193112 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" event={"ID":"b30158b4-a3dc-430b-820e-50d7f0160feb","Type":"ContainerDied","Data":"0cfc9275889b1d6f5570a0f64e2c905adf3c5291b20bada5a8ee9e120bf47b5f"} Nov 03 13:43:55.202158 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.193150 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cfc9275889b1d6f5570a0f64e2c905adf3c5291b20bada5a8ee9e120bf47b5f" Nov 03 13:43:55.205828 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.205794 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:43:55.272106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.262249 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-metrics-certs\") pod \"b30158b4-a3dc-430b-820e-50d7f0160feb\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " Nov 03 13:43:55.272106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.262350 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30158b4-a3dc-430b-820e-50d7f0160feb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30158b4-a3dc-430b-820e-50d7f0160feb-openshift-service-ca\") pod \"b30158b4-a3dc-430b-820e-50d7f0160feb\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " Nov 03 13:43:55.272106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.262427 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30158b4-a3dc-430b-820e-50d7f0160feb-kube-api-access-mkts9\" (UniqueName: \"kubernetes.io/projected/b30158b4-a3dc-430b-820e-50d7f0160feb-kube-api-access-mkts9\") pod \"b30158b4-a3dc-430b-820e-50d7f0160feb\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " Nov 03 13:43:55.272106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.262488 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls\") pod \"b30158b4-a3dc-430b-820e-50d7f0160feb\" (UID: \"b30158b4-a3dc-430b-820e-50d7f0160feb\") " Nov 03 13:43:55.272106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.265302 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30158b4-a3dc-430b-820e-50d7f0160feb-openshift-service-ca" pod "b30158b4-a3dc-430b-820e-50d7f0160feb" (UID: "b30158b4-a3dc-430b-820e-50d7f0160feb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:55.282633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.281772 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls" pod "b30158b4-a3dc-430b-820e-50d7f0160feb" (UID: "b30158b4-a3dc-430b-820e-50d7f0160feb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:55.282633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.282164 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-metrics-certs" pod "b30158b4-a3dc-430b-820e-50d7f0160feb" (UID: "b30158b4-a3dc-430b-820e-50d7f0160feb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:55.282633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.282332 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30158b4-a3dc-430b-820e-50d7f0160feb-kube-api-access-mkts9" pod "b30158b4-a3dc-430b-820e-50d7f0160feb" (UID: "b30158b4-a3dc-430b-820e-50d7f0160feb"). InnerVolumeSpecName "kube-api-access-mkts9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:55.364354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.364310 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:55.364354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.364352 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30158b4-a3dc-430b-820e-50d7f0160feb-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:55.364560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.364366 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkts9\" (UniqueName: \"kubernetes.io/projected/b30158b4-a3dc-430b-820e-50d7f0160feb-kube-api-access-mkts9\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:55.364560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.364381 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b30158b4-a3dc-430b-820e-50d7f0160feb-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:55.762390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.762351 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:55.894668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:55.894634 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:56.198946 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:56.198914 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq" Nov 03 13:43:56.381370 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:56.381090 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq"] Nov 03 13:43:56.425767 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:56.425733 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-007b0d-9bcbbfb7f-j9jkq"] Nov 03 13:43:56.951458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:56.951427 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30158b4-a3dc-430b-820e-50d7f0160feb" path="/var/lib/kubelet/pods/b30158b4-a3dc-430b-820e-50d7f0160feb/volumes" Nov 03 13:43:56.983135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:56.983103 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:57.153138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:57.153101 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:57.293985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:57.293910 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:58.680354 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:58.680320 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:43:59.195207 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:59.195172 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7"] Nov 03 13:43:59.195471 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:59.195444 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" containerID="cri-o://66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5" gracePeriod=30 Nov 03 13:43:59.232012 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:43:59.231984 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:03.893853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:03.893814 4924 patch_prober.go:29] interesting pod/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:44:03.894334 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:03.893881 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:44:04.465466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.465405 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:44:04.581779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.581746 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-istio-token\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-istio-token\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.581993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.581788 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-credential-socket\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.581993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.581901 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-envoy\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.581993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.581925 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-data\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.581993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.581949 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-certs\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.581993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.581982 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52de8c30-2906-4737-8dec-e81717588084-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de8c30-2906-4737-8dec-e81717588084-istiod-ca-cert\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.582257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582021 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-kube-api-access-bsjrg\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-kube-api-access-bsjrg\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.582257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582060 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-socket\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.582257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582166 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52de8c30-2906-4737-8dec-e81717588084-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de8c30-2906-4737-8dec-e81717588084-istio-podinfo\") pod \"52de8c30-2906-4737-8dec-e81717588084\" (UID: \"52de8c30-2906-4737-8dec-e81717588084\") " Nov 03 13:44:04.582257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582211 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-credential-socket" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:04.582462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582271 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-certs" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:04.582685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582628 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.582685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582655 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.582877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582681 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-data" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:04.582877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582726 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52de8c30-2906-4737-8dec-e81717588084-istiod-ca-cert" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:04.582877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.582802 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-socket" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:04.584262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.584238 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-envoy" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:04.584477 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.584454 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-istio-token" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:04.584546 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.584513 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52de8c30-2906-4737-8dec-e81717588084-istio-podinfo" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:04.584635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.584608 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-kube-api-access-bsjrg" pod "52de8c30-2906-4737-8dec-e81717588084" (UID: "52de8c30-2906-4737-8dec-e81717588084"). InnerVolumeSpecName "kube-api-access-bsjrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:04.684082 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.684025 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.684242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.684089 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.684242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.684108 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52de8c30-2906-4737-8dec-e81717588084-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.684242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.684128 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsjrg\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-kube-api-access-bsjrg\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.684242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.684144 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52de8c30-2906-4737-8dec-e81717588084-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.684242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.684161 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52de8c30-2906-4737-8dec-e81717588084-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.684242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.684176 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52de8c30-2906-4737-8dec-e81717588084-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:04.915523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:04.915481 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:05.259493 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.259397 4924 generic.go:356] "Generic (PLEG): container finished" podID="52de8c30-2906-4737-8dec-e81717588084" containerID="66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5" exitCode=0 Nov 03 13:44:05.259493 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.259445 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" Nov 03 13:44:05.259493 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.259477 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" event={"ID":"52de8c30-2906-4737-8dec-e81717588084","Type":"ContainerDied","Data":"66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5"} Nov 03 13:44:05.259774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.259532 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7" event={"ID":"52de8c30-2906-4737-8dec-e81717588084","Type":"ContainerDied","Data":"fc08ee6ca2abb746f639a1fae60516e43d6af3eb368cc8700c0c9ea69e70c8ca"} Nov 03 13:44:05.259774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.259560 4924 scope.go:122] "RemoveContainer" containerID="66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5" Nov 03 13:44:05.270040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.270016 4924 scope.go:122] "RemoveContainer" containerID="bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524" Nov 03 13:44:05.279857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.279832 4924 scope.go:122] "RemoveContainer" containerID="66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5" Nov 03 13:44:05.280208 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:44:05.280180 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5\": container with ID starting with 66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5 not found: ID does not exist" containerID="66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5" Nov 03 13:44:05.280278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.280221 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5"} err="failed to get container status \"66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5\": rpc error: code = NotFound desc = could not find container \"66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5\": container with ID starting with 66ebbb0bae06f91811e9c38db9a4594d51761e55846105000a4d2eb5b1643aa5 not found: ID does not exist" Nov 03 13:44:05.280278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.280246 4924 scope.go:122] "RemoveContainer" containerID="bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524" Nov 03 13:44:05.280594 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:44:05.280568 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524\": container with ID starting with bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524 not found: ID does not exist" containerID="bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524" Nov 03 13:44:05.280672 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.280605 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524"} err="failed to get container status \"bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524\": rpc error: code = NotFound desc = could not find container \"bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524\": container with ID starting with bbeadc704f1b1da45fc03ee89f9893643e464e04c5ff89633556ae7e9b66a524 not found: ID does not exist" Nov 03 13:44:05.467727 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.467687 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7"] Nov 03 13:44:05.514382 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:05.514291 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-007b0d-openshift-default-6599867f97-b9cw7"] Nov 03 13:44:06.952684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:06.952598 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52de8c30-2906-4737-8dec-e81717588084" path="/var/lib/kubelet/pods/52de8c30-2906-4737-8dec-e81717588084/volumes" Nov 03 13:44:10.073252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:10.073202 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:15.329736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:15.329701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:18.561647 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:44:18.561607 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:44:20.501308 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:20.501276 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:25.982984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:25.982947 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:27.329002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:27.328974 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:44:27.331579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:27.331552 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:44:27.386298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:27.386258 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:30.074378 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:30.074343 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:30.258115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:30.258040 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:32.339589 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:32.339550 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5"] Nov 03 13:44:32.442507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:32.442466 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:33.002561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:33.002528 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:38.249561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:38.249518 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:43.046161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:43.046123 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:43.587551 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:43.587512 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:43.757015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:43.756978 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:43.971396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:43.971357 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:45.010976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:45.010944 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:47.077338 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.077298 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:47.359911 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.359803 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" podUID="26603eeb-8c51-4378-bdeb-3774e2455395" containerName="main" containerID="cri-o://21f33220c572d1a45277306fd05afe580058be3d51adfe99302cceb27c027e2d" gracePeriod=45 Nov 03 13:44:47.442800 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.442755 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" podUID="26603eeb-8c51-4378-bdeb-3774e2455395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.106:8000/ready\": dial tcp 10.132.0.106:8000: connect: connection refused" Nov 03 13:44:47.557869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.557195 4924 generic.go:356] "Generic (PLEG): container finished" podID="26603eeb-8c51-4378-bdeb-3774e2455395" containerID="21f33220c572d1a45277306fd05afe580058be3d51adfe99302cceb27c027e2d" exitCode=0 Nov 03 13:44:47.557869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.557266 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" event={"ID":"26603eeb-8c51-4378-bdeb-3774e2455395","Type":"ContainerDied","Data":"21f33220c572d1a45277306fd05afe580058be3d51adfe99302cceb27c027e2d"} Nov 03 13:44:47.618291 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.618226 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:44:47.674703 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.674662 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26603eeb-8c51-4378-bdeb-3774e2455395-tls-certs\" (UniqueName: \"kubernetes.io/secret/26603eeb-8c51-4378-bdeb-3774e2455395-tls-certs\") pod \"26603eeb-8c51-4378-bdeb-3774e2455395\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " Nov 03 13:44:47.674874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.674750 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-tmp-dir\") pod \"26603eeb-8c51-4378-bdeb-3774e2455395\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " Nov 03 13:44:47.674874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.674799 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26603eeb-8c51-4378-bdeb-3774e2455395-kube-api-access-9w8g6\" (UniqueName: \"kubernetes.io/projected/26603eeb-8c51-4378-bdeb-3774e2455395-kube-api-access-9w8g6\") pod \"26603eeb-8c51-4378-bdeb-3774e2455395\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " Nov 03 13:44:47.674874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.674862 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-home\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-home\") pod \"26603eeb-8c51-4378-bdeb-3774e2455395\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " Nov 03 13:44:47.675040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.674913 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-dshm\") pod \"26603eeb-8c51-4378-bdeb-3774e2455395\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " Nov 03 13:44:47.675040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.674963 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-model-cache\") pod \"26603eeb-8c51-4378-bdeb-3774e2455395\" (UID: \"26603eeb-8c51-4378-bdeb-3774e2455395\") " Nov 03 13:44:47.675040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.675027 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-tmp-dir" pod "26603eeb-8c51-4378-bdeb-3774e2455395" (UID: "26603eeb-8c51-4378-bdeb-3774e2455395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.675278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.675233 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-home" pod "26603eeb-8c51-4378-bdeb-3774e2455395" (UID: "26603eeb-8c51-4378-bdeb-3774e2455395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.675380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.675346 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-model-cache" pod "26603eeb-8c51-4378-bdeb-3774e2455395" (UID: "26603eeb-8c51-4378-bdeb-3774e2455395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.675505 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.675460 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:47.675505 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.675483 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:47.675505 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.675497 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:47.677290 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.677262 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26603eeb-8c51-4378-bdeb-3774e2455395-kube-api-access-9w8g6" pod "26603eeb-8c51-4378-bdeb-3774e2455395" (UID: "26603eeb-8c51-4378-bdeb-3774e2455395"). InnerVolumeSpecName "kube-api-access-9w8g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:47.677399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.677290 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-dshm" pod "26603eeb-8c51-4378-bdeb-3774e2455395" (UID: "26603eeb-8c51-4378-bdeb-3774e2455395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.677399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.677300 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26603eeb-8c51-4378-bdeb-3774e2455395-tls-certs" pod "26603eeb-8c51-4378-bdeb-3774e2455395" (UID: "26603eeb-8c51-4378-bdeb-3774e2455395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:47.776585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.776552 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w8g6\" (UniqueName: \"kubernetes.io/projected/26603eeb-8c51-4378-bdeb-3774e2455395-kube-api-access-9w8g6\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:47.776585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.776581 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26603eeb-8c51-4378-bdeb-3774e2455395-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:47.776585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:47.776591 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26603eeb-8c51-4378-bdeb-3774e2455395-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.559867 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:44:48.559824 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:44:48.565913 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:48.565889 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" Nov 03 13:44:48.566050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:48.565925 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5" event={"ID":"26603eeb-8c51-4378-bdeb-3774e2455395","Type":"ContainerDied","Data":"fa9299ea17543892d425cb645f8df160529e4909f0ba7071385793994374ae05"} Nov 03 13:44:48.566050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:48.565975 4924 scope.go:122] "RemoveContainer" containerID="21f33220c572d1a45277306fd05afe580058be3d51adfe99302cceb27c027e2d" Nov 03 13:44:48.731982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:48.731946 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5"] Nov 03 13:44:48.747567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:48.747515 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:48.779169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:48.779133 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a770b9/ipp-route-a770b9-kserve-75fbf6fcb9-xpcg5"] Nov 03 13:44:48.951523 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:48.951483 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26603eeb-8c51-4378-bdeb-3774e2455395" path="/var/lib/kubelet/pods/26603eeb-8c51-4378-bdeb-3774e2455395/volumes" Nov 03 13:44:48.990746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:48.990679 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:50.463202 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:50.463162 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:50.643608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:50.643568 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:51.083200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:51.083165 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b"] Nov 03 13:44:51.083428 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:51.083400 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" containerID="cri-o://c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e" gracePeriod=30 Nov 03 13:44:51.106536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:51.106500 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:52.657476 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:52.657446 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:53.217742 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.217699 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k"] Nov 03 13:44:53.218481 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218453 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" Nov 03 13:44:53.218481 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218481 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218507 4924 cpu_manager.go:414] "containerMap: removing container" podUID="b30158b4-a3dc-430b-820e-50d7f0160feb" containerName="maas-api" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218518 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30158b4-a3dc-430b-820e-50d7f0160feb" containerName="maas-api" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218542 4924 cpu_manager.go:414] "containerMap: removing container" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218551 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218568 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218575 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218594 4924 cpu_manager.go:414] "containerMap: removing container" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218601 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218612 4924 cpu_manager.go:414] "containerMap: removing container" podUID="26603eeb-8c51-4378-bdeb-3774e2455395" containerName="main" Nov 03 13:44:53.218723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218620 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26603eeb-8c51-4378-bdeb-3774e2455395" containerName="main" Nov 03 13:44:53.219111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218785 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" Nov 03 13:44:53.219111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218799 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" Nov 03 13:44:53.219111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218810 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" Nov 03 13:44:53.219111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218821 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" Nov 03 13:44:53.219111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218829 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="26603eeb-8c51-4378-bdeb-3774e2455395" containerName="main" Nov 03 13:44:53.219111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218846 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77521ec-beb3-4383-b2e2-e2bb66557174" containerName="istio-proxy" Nov 03 13:44:53.219111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.218857 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30158b4-a3dc-430b-820e-50d7f0160feb" containerName="maas-api" Nov 03 13:44:53.223343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.223320 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.304093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.303363 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k"] Nov 03 13:44:53.304093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.303597 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-977f32-openshift-default-dockercfg-n5rqq\"" Nov 03 13:44:53.342757 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.342721 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-envoy\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.342970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.342775 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-certs\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.342970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.342935 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istiod-ca-cert\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.343112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.343007 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dfp\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-kube-api-access-b4dfp\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.343160 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.343137 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-token\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.343268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.343247 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-socket\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.343309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.343298 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-data\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.343364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.343349 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-podinfo\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.343486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.343461 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-credential-socket\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.382293 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.381541 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:53.444508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444467 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-credential-socket\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.444723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444529 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-envoy\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.444723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444562 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-certs\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.444723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444641 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istiod-ca-cert\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.444723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444696 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dfp\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-kube-api-access-b4dfp\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.444938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444738 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-token\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.444938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444799 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-socket\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.444938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444838 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-data\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.444938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.444865 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-podinfo\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.445915 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.445881 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istiod-ca-cert\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.446231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.446181 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-certs\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.446947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.446921 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-credential-socket\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.447238 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.447195 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-socket\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.447455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.447433 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-data\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.448658 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.448629 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-podinfo\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.467471 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.467435 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-envoy\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.504661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.504600 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dfp\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-kube-api-access-b4dfp\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.504804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.504663 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-token\") pod \"e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.554259 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.554218 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:53.837384 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.837299 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:53.987669 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.987640 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k"] Nov 03 13:44:53.989896 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:44:53.989867 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5266ad13_5565_49e7_8aed_c6d9ed2ea610.slice/crio-9605d379c81160fdec5dde4792dc60b8518826e43da0f0eea2a465c34d806263 WatchSource:0}: Error finding container 9605d379c81160fdec5dde4792dc60b8518826e43da0f0eea2a465c34d806263: Status 404 returned error can't find the container with id 9605d379c81160fdec5dde4792dc60b8518826e43da0f0eea2a465c34d806263 Nov 03 13:44:53.992752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.992708 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:44:53.992832 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.992809 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:44:53.992871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:53.992858 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:44:54.610124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:54.610054 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" event={"ID":"5266ad13-5565-49e7-8aed-c6d9ed2ea610","Type":"ContainerStarted","Data":"f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9"} Nov 03 13:44:54.610314 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:54.610180 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" event={"ID":"5266ad13-5565-49e7-8aed-c6d9ed2ea610","Type":"ContainerStarted","Data":"9605d379c81160fdec5dde4792dc60b8518826e43da0f0eea2a465c34d806263"} Nov 03 13:44:54.730879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:54.730810 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" podStartSLOduration=1.730791293 podStartE2EDuration="1.730791293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:44:54.727329166 +0000 UTC m=+2427.921732760" watchObservedRunningTime="2026-11-03 13:44:54.730791293 +0000 UTC m=+2427.925194888" Nov 03 13:44:54.912325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:54.912287 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:55.094493 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:55.094452 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:55.219560 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:55.219458 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:55.555012 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:55.554917 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:55.560754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:55.560725 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:55.615978 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:55.615948 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:55.616902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:55.616881 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:44:56.386879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.386498 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:44:56.440533 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.440495 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:56.498944 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.498907 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-data\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.498969 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-envoy\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499007 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-credential-socket\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499115 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-certs\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499143 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-socket\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499183 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-kube-api-access-n2zks\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-kube-api-access-n2zks\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499220 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1d972d-d193-4677-a9bb-9b259b8d0bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1d972d-d193-4677-a9bb-9b259b8d0bca-istiod-ca-cert\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499256 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-podinfo\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499279 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-data" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499317 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-token\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-token\") pod \"be1d972d-d193-4677-a9bb-9b259b8d0bca\" (UID: \"be1d972d-d193-4677-a9bb-9b259b8d0bca\") " Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499319 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-credential-socket" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499395 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-socket" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499407 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-certs" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499618 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1d972d-d193-4677-a9bb-9b259b8d0bca-istiod-ca-cert" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499870 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499888 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499904 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499920 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.499947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.499934 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be1d972d-d193-4677-a9bb-9b259b8d0bca-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.502105 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.502024 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-podinfo" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:56.502105 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.502042 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-envoy" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:56.502658 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.502634 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-kube-api-access-n2zks" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "kube-api-access-n2zks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:56.502962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.502925 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-token" pod "be1d972d-d193-4677-a9bb-9b259b8d0bca" (UID: "be1d972d-d193-4677-a9bb-9b259b8d0bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:56.600759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.600729 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.600759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.600758 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2zks\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-kube-api-access-n2zks\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.600964 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.600770 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.600964 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.600778 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be1d972d-d193-4677-a9bb-9b259b8d0bca-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:56.623781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.623749 4924 generic.go:356] "Generic (PLEG): container finished" podID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerID="c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e" exitCode=0 Nov 03 13:44:56.623938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.623828 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" Nov 03 13:44:56.623938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.623832 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" event={"ID":"be1d972d-d193-4677-a9bb-9b259b8d0bca","Type":"ContainerDied","Data":"c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e"} Nov 03 13:44:56.623938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.623885 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b" event={"ID":"be1d972d-d193-4677-a9bb-9b259b8d0bca","Type":"ContainerDied","Data":"7e195ff6580fcc0abb74181768f72c4ace49fc14a43fb43859177037d446cd6f"} Nov 03 13:44:56.623938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.623908 4924 scope.go:122] "RemoveContainer" containerID="c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e" Nov 03 13:44:56.636294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.636277 4924 scope.go:122] "RemoveContainer" containerID="697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17" Nov 03 13:44:56.645829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.645808 4924 scope.go:122] "RemoveContainer" containerID="c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e" Nov 03 13:44:56.646165 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:44:56.646145 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e\": container with ID starting with c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e not found: ID does not exist" containerID="c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e" Nov 03 13:44:56.646275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.646175 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e"} err="failed to get container status \"c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e\": rpc error: code = NotFound desc = could not find container \"c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e\": container with ID starting with c3e476afd98ffda27557bb5886144eca5ce4bc7bbc079dd39c0739ccd83b659e not found: ID does not exist" Nov 03 13:44:56.646275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.646192 4924 scope.go:122] "RemoveContainer" containerID="697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17" Nov 03 13:44:56.646461 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:44:56.646435 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17\": container with ID starting with 697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17 not found: ID does not exist" containerID="697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17" Nov 03 13:44:56.646515 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.646472 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17"} err="failed to get container status \"697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17\": rpc error: code = NotFound desc = could not find container \"697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17\": container with ID starting with 697c6e1e9d14469bebafe9085246337be85a9fb8dafab7299e2e5a94da6e9c17 not found: ID does not exist" Nov 03 13:44:56.793434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.793393 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b"] Nov 03 13:44:56.797818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.797789 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6bd5c1-openshift-default-7547c954c-v2k7b"] Nov 03 13:44:56.953316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:56.953279 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" path="/var/lib/kubelet/pods/be1d972d-d193-4677-a9bb-9b259b8d0bca/volumes" Nov 03 13:44:57.849335 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:57.849295 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:58.138426 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:58.138341 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:44:58.370561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:44:58.370529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:00.227535 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.227494 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd"] Nov 03 13:45:00.228043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228028 4924 cpu_manager.go:414] "containerMap: removing container" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" Nov 03 13:45:00.228111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228044 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" Nov 03 13:45:00.228111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228054 4924 cpu_manager.go:414] "containerMap: removing container" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" Nov 03 13:45:00.228111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228059 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" Nov 03 13:45:00.228111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228089 4924 cpu_manager.go:414] "containerMap: removing container" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" Nov 03 13:45:00.228111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228095 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16c9379-dc86-4ecd-a669-ebe078128f6f" containerName="istio-proxy" Nov 03 13:45:00.228111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228103 4924 cpu_manager.go:414] "containerMap: removing container" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" Nov 03 13:45:00.228111 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228107 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" Nov 03 13:45:00.228344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228193 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="52de8c30-2906-4737-8dec-e81717588084" containerName="istio-proxy" Nov 03 13:45:00.228344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228201 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" Nov 03 13:45:00.228344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.228212 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1d972d-d193-4677-a9bb-9b259b8d0bca" containerName="istio-proxy" Nov 03 13:45:00.233340 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.233318 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:00.269483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.269448 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp"] Nov 03 13:45:00.273882 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.273858 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:00.289563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.289530 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j"] Nov 03 13:45:00.294824 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.294773 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:00.295636 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.295608 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd"] Nov 03 13:45:00.339126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.339091 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp"] Nov 03 13:45:00.342320 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.342281 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd\" (UID: \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:00.343268 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.343146 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cg7\" (UniqueName: \"kubernetes.io/projected/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-kube-api-access-j2cg7\") pod \"maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd\" (UID: \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:00.352506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.352479 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:45:00.403701 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.403664 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j"] Nov 03 13:45:00.444225 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.444186 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgtj\" (UniqueName: \"kubernetes.io/projected/0eafde52-d648-44db-a902-9394caf28cff-kube-api-access-xhgtj\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp\" (UID: \"0eafde52-d648-44db-a902-9394caf28cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:00.444398 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.444256 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpghp\" (UniqueName: \"kubernetes.io/projected/ac5c3559-aa52-468d-9553-01a2b6f2498b-kube-api-access-wpghp\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j\" (UID: \"ac5c3559-aa52-468d-9553-01a2b6f2498b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:00.444398 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.444364 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd\" (UID: \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:00.444491 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.444415 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c3559-aa52-468d-9553-01a2b6f2498b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j\" (UID: \"ac5c3559-aa52-468d-9553-01a2b6f2498b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:00.444534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.444495 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cg7\" (UniqueName: \"kubernetes.io/projected/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-kube-api-access-j2cg7\") pod \"maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd\" (UID: \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:00.444595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.444571 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eafde52-d648-44db-a902-9394caf28cff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp\" (UID: \"0eafde52-d648-44db-a902-9394caf28cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:00.445049 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.445032 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd\" (UID: \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:00.500528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.500442 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cg7\" (UniqueName: \"kubernetes.io/projected/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-kube-api-access-j2cg7\") pod \"maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd\" (UID: \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:00.546271 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.546227 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c3559-aa52-468d-9553-01a2b6f2498b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j\" (UID: \"ac5c3559-aa52-468d-9553-01a2b6f2498b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:00.546447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.546360 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eafde52-d648-44db-a902-9394caf28cff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp\" (UID: \"0eafde52-d648-44db-a902-9394caf28cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:00.546521 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.546453 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgtj\" (UniqueName: \"kubernetes.io/projected/0eafde52-d648-44db-a902-9394caf28cff-kube-api-access-xhgtj\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp\" (UID: \"0eafde52-d648-44db-a902-9394caf28cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:00.546604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.546556 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpghp\" (UniqueName: \"kubernetes.io/projected/ac5c3559-aa52-468d-9553-01a2b6f2498b-kube-api-access-wpghp\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j\" (UID: \"ac5c3559-aa52-468d-9553-01a2b6f2498b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:00.547016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.546990 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c3559-aa52-468d-9553-01a2b6f2498b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j\" (UID: \"ac5c3559-aa52-468d-9553-01a2b6f2498b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:00.547140 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.547035 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eafde52-d648-44db-a902-9394caf28cff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp\" (UID: \"0eafde52-d648-44db-a902-9394caf28cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:00.547834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.547818 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:00.606061 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.606024 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgtj\" (UniqueName: \"kubernetes.io/projected/0eafde52-d648-44db-a902-9394caf28cff-kube-api-access-xhgtj\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp\" (UID: \"0eafde52-d648-44db-a902-9394caf28cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:00.607464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.607434 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpghp\" (UniqueName: \"kubernetes.io/projected/ac5c3559-aa52-468d-9553-01a2b6f2498b-kube-api-access-wpghp\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j\" (UID: \"ac5c3559-aa52-468d-9553-01a2b6f2498b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:00.612302 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.612273 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:00.891505 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.891436 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:00.925459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:00.925428 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd"] Nov 03 13:45:00.926616 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:45:00.926587 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd6d01a_ed64_4ae9_ab1f_8897bef9aa91.slice/crio-52568a05de0a971203b2430ccb0c0b4daa45e3e2030ab3bd7a7a4baef46f8e65 WatchSource:0}: Error finding container 52568a05de0a971203b2430ccb0c0b4daa45e3e2030ab3bd7a7a4baef46f8e65: Status 404 returned error can't find the container with id 52568a05de0a971203b2430ccb0c0b4daa45e3e2030ab3bd7a7a4baef46f8e65 Nov 03 13:45:01.036735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.036708 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j"] Nov 03 13:45:01.039475 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:45:01.039446 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5c3559_aa52_468d_9553_01a2b6f2498b.slice/crio-2c6157c17f58b4f5f683a2327ee4f609fba255e3dfed61b35f4ccc8da47617f1 WatchSource:0}: Error finding container 2c6157c17f58b4f5f683a2327ee4f609fba255e3dfed61b35f4ccc8da47617f1: Status 404 returned error can't find the container with id 2c6157c17f58b4f5f683a2327ee4f609fba255e3dfed61b35f4ccc8da47617f1 Nov 03 13:45:01.272532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.272502 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:01.282524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.282500 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp"] Nov 03 13:45:01.284271 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:45:01.284243 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eafde52_d648_44db_a902_9394caf28cff.slice/crio-dd332e0bb4dcb45a6690160e39338ebd39c9bc260d4ed85a1d4965654aafefd7 WatchSource:0}: Error finding container dd332e0bb4dcb45a6690160e39338ebd39c9bc260d4ed85a1d4965654aafefd7: Status 404 returned error can't find the container with id dd332e0bb4dcb45a6690160e39338ebd39c9bc260d4ed85a1d4965654aafefd7 Nov 03 13:45:01.532278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.532245 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:01.662658 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.662621 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" event={"ID":"0eafde52-d648-44db-a902-9394caf28cff","Type":"ContainerStarted","Data":"5fd2b5e4b3c1e7714de8441eaf2778050bfa123eb8703cc90952a8b670c9fe5c"} Nov 03 13:45:01.662819 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.662689 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" event={"ID":"0eafde52-d648-44db-a902-9394caf28cff","Type":"ContainerStarted","Data":"dd332e0bb4dcb45a6690160e39338ebd39c9bc260d4ed85a1d4965654aafefd7"} Nov 03 13:45:01.664297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.664258 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" event={"ID":"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91","Type":"ContainerStarted","Data":"c3b638b3ca91052440405d4de2c706be84cbdc15e48d8cc701674af7339c4e05"} Nov 03 13:45:01.664410 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.664315 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" event={"ID":"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91","Type":"ContainerStarted","Data":"52568a05de0a971203b2430ccb0c0b4daa45e3e2030ab3bd7a7a4baef46f8e65"} Nov 03 13:45:01.665850 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.665823 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" event={"ID":"ac5c3559-aa52-468d-9553-01a2b6f2498b","Type":"ContainerStarted","Data":"25dfc6696c13916a1f70a7adb17d51dea908481d2f8c87e6f60cbf288133d4c1"} Nov 03 13:45:01.665948 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.665871 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" event={"ID":"ac5c3559-aa52-468d-9553-01a2b6f2498b","Type":"ContainerStarted","Data":"2c6157c17f58b4f5f683a2327ee4f609fba255e3dfed61b35f4ccc8da47617f1"} Nov 03 13:45:01.784046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.783944 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" podStartSLOduration=1.783927105 podStartE2EDuration="1.783927105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.783235987 +0000 UTC m=+2434.977639592" watchObservedRunningTime="2026-11-03 13:45:01.783927105 +0000 UTC m=+2434.978330699" Nov 03 13:45:01.906655 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:01.906592 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" podStartSLOduration=1.9065726729999999 podStartE2EDuration="1.906572673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.906240374 +0000 UTC m=+2435.100643968" watchObservedRunningTime="2026-11-03 13:45:01.906572673 +0000 UTC m=+2435.100976262" Nov 03 13:45:02.029992 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.029873 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" podStartSLOduration=2.029853763 podStartE2EDuration="2.029853763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:02.02757092 +0000 UTC m=+2435.221974515" watchObservedRunningTime="2026-11-03 13:45:02.029853763 +0000 UTC m=+2435.224257359" Nov 03 13:45:02.109910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.109872 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65"] Nov 03 13:45:02.114042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.114017 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.168836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.168798 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ba6fbd-serving-cert\"" Nov 03 13:45:02.181197 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.181155 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65"] Nov 03 13:45:02.264847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.264810 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.264847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.264861 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-metrics-certs\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.265130 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.265038 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlslm\" (UniqueName: \"kubernetes.io/projected/e209e423-2c61-428e-beae-bbc476dc2e7e-kube-api-access-qlslm\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.265130 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.265115 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e209e423-2c61-428e-beae-bbc476dc2e7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.343720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.343635 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:02.368793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.367035 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.368793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.367122 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-metrics-certs\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.368793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.367194 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlslm\" (UniqueName: \"kubernetes.io/projected/e209e423-2c61-428e-beae-bbc476dc2e7e-kube-api-access-qlslm\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.368793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.367222 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e209e423-2c61-428e-beae-bbc476dc2e7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.368793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.368048 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e209e423-2c61-428e-beae-bbc476dc2e7e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.368793 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:02.368172 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-serving-cert: secret "maas-api-e2e-praxis-b-ba6fbd-serving-cert" not found Nov 03 13:45:02.368793 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:02.368238 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls podName:e209e423-2c61-428e-beae-bbc476dc2e7e nodeName:}" failed. No retries permitted until 2026-11-03 13:45:02.86821751 +0000 UTC m=+2436.062621087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls") pod "maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" (UID: "e209e423-2c61-428e-beae-bbc476dc2e7e") : secret "maas-api-e2e-praxis-b-ba6fbd-serving-cert" not found Nov 03 13:45:02.379719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.379680 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-metrics-certs\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.424940 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.424910 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlslm\" (UniqueName: \"kubernetes.io/projected/e209e423-2c61-428e-beae-bbc476dc2e7e-kube-api-access-qlslm\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.575511 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.575476 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:02.679914 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.679861 4924 generic.go:356] "Generic (PLEG): container finished" podID="0eafde52-d648-44db-a902-9394caf28cff" containerID="5fd2b5e4b3c1e7714de8441eaf2778050bfa123eb8703cc90952a8b670c9fe5c" exitCode=60 Nov 03 13:45:02.680277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.679954 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" event={"ID":"0eafde52-d648-44db-a902-9394caf28cff","Type":"ContainerDied","Data":"5fd2b5e4b3c1e7714de8441eaf2778050bfa123eb8703cc90952a8b670c9fe5c"} Nov 03 13:45:02.680409 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.680388 4924 scope.go:122] "RemoveContainer" containerID="5fd2b5e4b3c1e7714de8441eaf2778050bfa123eb8703cc90952a8b670c9fe5c" Nov 03 13:45:02.683973 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.683948 4924 generic.go:356] "Generic (PLEG): container finished" podID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerID="c3b638b3ca91052440405d4de2c706be84cbdc15e48d8cc701674af7339c4e05" exitCode=60 Nov 03 13:45:02.684124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.684036 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" event={"ID":"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91","Type":"ContainerDied","Data":"c3b638b3ca91052440405d4de2c706be84cbdc15e48d8cc701674af7339c4e05"} Nov 03 13:45:02.684357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.684249 4924 scope.go:122] "RemoveContainer" containerID="c3b638b3ca91052440405d4de2c706be84cbdc15e48d8cc701674af7339c4e05" Nov 03 13:45:02.706982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.706954 4924 generic.go:356] "Generic (PLEG): container finished" podID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerID="25dfc6696c13916a1f70a7adb17d51dea908481d2f8c87e6f60cbf288133d4c1" exitCode=60 Nov 03 13:45:02.707146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.707019 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" event={"ID":"ac5c3559-aa52-468d-9553-01a2b6f2498b","Type":"ContainerDied","Data":"25dfc6696c13916a1f70a7adb17d51dea908481d2f8c87e6f60cbf288133d4c1"} Nov 03 13:45:02.707633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.707608 4924 scope.go:122] "RemoveContainer" containerID="25dfc6696c13916a1f70a7adb17d51dea908481d2f8c87e6f60cbf288133d4c1" Nov 03 13:45:02.749854 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.749790 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:02.879094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.875544 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:02.929593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:02.929262 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:03.028972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.028931 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:03.482641 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.482606 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65"] Nov 03 13:45:03.483478 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:45:03.483445 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode209e423_2c61_428e_beae_bbc476dc2e7e.slice/crio-811776ed291b39cf87616ff74ce8f9622c19c63f8efcbb5c6d23bbe0228051d4 WatchSource:0}: Error finding container 811776ed291b39cf87616ff74ce8f9622c19c63f8efcbb5c6d23bbe0228051d4: Status 404 returned error can't find the container with id 811776ed291b39cf87616ff74ce8f9622c19c63f8efcbb5c6d23bbe0228051d4 Nov 03 13:45:03.716627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.716587 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" event={"ID":"0eafde52-d648-44db-a902-9394caf28cff","Type":"ContainerStarted","Data":"976dfd279a415820e18351c0212b1837a011ce2d8abb4678de543d06c7a7a2a7"} Nov 03 13:45:03.718475 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.718447 4924 generic.go:356] "Generic (PLEG): container finished" podID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerID="e0927236601b4fc7552b997043c959ae1edaa22429d3e2ee37bdb382dce4b439" exitCode=60 Nov 03 13:45:03.718622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.718515 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" event={"ID":"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91","Type":"ContainerDied","Data":"e0927236601b4fc7552b997043c959ae1edaa22429d3e2ee37bdb382dce4b439"} Nov 03 13:45:03.718622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.718582 4924 scope.go:122] "RemoveContainer" containerID="c3b638b3ca91052440405d4de2c706be84cbdc15e48d8cc701674af7339c4e05" Nov 03 13:45:03.718849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.718826 4924 scope.go:122] "RemoveContainer" containerID="e0927236601b4fc7552b997043c959ae1edaa22429d3e2ee37bdb382dce4b439" Nov 03 13:45:03.719165 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:03.719128 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd_odh-ai-gateway-infra(bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" Nov 03 13:45:03.719983 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.719963 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" event={"ID":"e209e423-2c61-428e-beae-bbc476dc2e7e","Type":"ContainerStarted","Data":"811776ed291b39cf87616ff74ce8f9622c19c63f8efcbb5c6d23bbe0228051d4"} Nov 03 13:45:03.721821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:03.721797 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" event={"ID":"ac5c3559-aa52-468d-9553-01a2b6f2498b","Type":"ContainerStarted","Data":"8f5627fd315d072f78c0205dff691d0e41f4f60ff25072fdf925dcd1f2fa987c"} Nov 03 13:45:04.729826 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.729740 4924 generic.go:356] "Generic (PLEG): container finished" podID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerID="8f5627fd315d072f78c0205dff691d0e41f4f60ff25072fdf925dcd1f2fa987c" exitCode=60 Nov 03 13:45:04.730297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.729891 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" event={"ID":"ac5c3559-aa52-468d-9553-01a2b6f2498b","Type":"ContainerDied","Data":"8f5627fd315d072f78c0205dff691d0e41f4f60ff25072fdf925dcd1f2fa987c"} Nov 03 13:45:04.730297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.729962 4924 scope.go:122] "RemoveContainer" containerID="25dfc6696c13916a1f70a7adb17d51dea908481d2f8c87e6f60cbf288133d4c1" Nov 03 13:45:04.730297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.730164 4924 scope.go:122] "RemoveContainer" containerID="8f5627fd315d072f78c0205dff691d0e41f4f60ff25072fdf925dcd1f2fa987c" Nov 03 13:45:04.730497 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:04.730467 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j_odh-ai-gateway-infra(ac5c3559-aa52-468d-9553-01a2b6f2498b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" Nov 03 13:45:04.732150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.732128 4924 generic.go:356] "Generic (PLEG): container finished" podID="0eafde52-d648-44db-a902-9394caf28cff" containerID="976dfd279a415820e18351c0212b1837a011ce2d8abb4678de543d06c7a7a2a7" exitCode=60 Nov 03 13:45:04.732260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.732194 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" event={"ID":"0eafde52-d648-44db-a902-9394caf28cff","Type":"ContainerDied","Data":"976dfd279a415820e18351c0212b1837a011ce2d8abb4678de543d06c7a7a2a7"} Nov 03 13:45:04.732504 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.732487 4924 scope.go:122] "RemoveContainer" containerID="976dfd279a415820e18351c0212b1837a011ce2d8abb4678de543d06c7a7a2a7" Nov 03 13:45:04.732788 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:04.732767 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp_odh-ai-gateway-infra(0eafde52-d648-44db-a902-9394caf28cff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" podUID="0eafde52-d648-44db-a902-9394caf28cff" Nov 03 13:45:04.734082 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.734046 4924 scope.go:122] "RemoveContainer" containerID="e0927236601b4fc7552b997043c959ae1edaa22429d3e2ee37bdb382dce4b439" Nov 03 13:45:04.734352 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:04.734328 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd_odh-ai-gateway-infra(bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" Nov 03 13:45:04.735464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.735437 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" event={"ID":"e209e423-2c61-428e-beae-bbc476dc2e7e","Type":"ContainerStarted","Data":"a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7"} Nov 03 13:45:04.735594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.735577 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:04.744195 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.744167 4924 scope.go:122] "RemoveContainer" containerID="5fd2b5e4b3c1e7714de8441eaf2778050bfa123eb8703cc90952a8b670c9fe5c" Nov 03 13:45:04.795212 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:04.795174 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:05.091131 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:05.091014 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" podStartSLOduration=2.123655282 podStartE2EDuration="3.090995559s" totalImagesPullingTime="967.340277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:05.088895852 +0000 UTC m=+2438.283299466" watchObservedRunningTime="2026-11-03 13:45:05.090995559 +0000 UTC m=+2438.285399154" Nov 03 13:45:05.356356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:05.356234 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:05.747442 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:05.747401 4924 scope.go:122] "RemoveContainer" containerID="976dfd279a415820e18351c0212b1837a011ce2d8abb4678de543d06c7a7a2a7" Nov 03 13:45:05.747935 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:05.747713 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp_odh-ai-gateway-infra(0eafde52-d648-44db-a902-9394caf28cff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" podUID="0eafde52-d648-44db-a902-9394caf28cff" Nov 03 13:45:05.749107 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:05.749059 4924 scope.go:122] "RemoveContainer" containerID="8f5627fd315d072f78c0205dff691d0e41f4f60ff25072fdf925dcd1f2fa987c" Nov 03 13:45:05.749330 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:05.749311 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j_odh-ai-gateway-infra(ac5c3559-aa52-468d-9553-01a2b6f2498b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" Nov 03 13:45:08.163908 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:08.163872 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:10.754462 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:10.754431 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:45:15.168468 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:15.168428 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:15.943746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:15.943699 4924 scope.go:122] "RemoveContainer" containerID="976dfd279a415820e18351c0212b1837a011ce2d8abb4678de543d06c7a7a2a7" Nov 03 13:45:16.825352 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:16.825316 4924 generic.go:356] "Generic (PLEG): container finished" podID="0eafde52-d648-44db-a902-9394caf28cff" containerID="e9ed25d5a880d844048a9301a53d200ec589bc5aade398b8dd41b554a29714ed" exitCode=60 Nov 03 13:45:16.825762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:16.825387 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" event={"ID":"0eafde52-d648-44db-a902-9394caf28cff","Type":"ContainerDied","Data":"e9ed25d5a880d844048a9301a53d200ec589bc5aade398b8dd41b554a29714ed"} Nov 03 13:45:16.825762 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:16.825447 4924 scope.go:122] "RemoveContainer" containerID="976dfd279a415820e18351c0212b1837a011ce2d8abb4678de543d06c7a7a2a7" Nov 03 13:45:16.825853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:16.825796 4924 scope.go:122] "RemoveContainer" containerID="e9ed25d5a880d844048a9301a53d200ec589bc5aade398b8dd41b554a29714ed" Nov 03 13:45:16.826151 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:16.826128 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp_odh-ai-gateway-infra(0eafde52-d648-44db-a902-9394caf28cff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" podUID="0eafde52-d648-44db-a902-9394caf28cff" Nov 03 13:45:16.943470 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:16.943441 4924 scope.go:122] "RemoveContainer" containerID="e0927236601b4fc7552b997043c959ae1edaa22429d3e2ee37bdb382dce4b439" Nov 03 13:45:16.943662 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:16.943549 4924 scope.go:122] "RemoveContainer" containerID="8f5627fd315d072f78c0205dff691d0e41f4f60ff25072fdf925dcd1f2fa987c" Nov 03 13:45:17.836406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:17.836372 4924 generic.go:356] "Generic (PLEG): container finished" podID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerID="66c8a983e5bae7f7b44ba081c11badb26ed6c0da56aa5e90bb6d4c8f2c09e5b9" exitCode=60 Nov 03 13:45:17.836844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:17.836437 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" event={"ID":"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91","Type":"ContainerDied","Data":"66c8a983e5bae7f7b44ba081c11badb26ed6c0da56aa5e90bb6d4c8f2c09e5b9"} Nov 03 13:45:17.836844 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:17.836494 4924 scope.go:122] "RemoveContainer" containerID="e0927236601b4fc7552b997043c959ae1edaa22429d3e2ee37bdb382dce4b439" Nov 03 13:45:17.836963 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:17.836885 4924 scope.go:122] "RemoveContainer" containerID="66c8a983e5bae7f7b44ba081c11badb26ed6c0da56aa5e90bb6d4c8f2c09e5b9" Nov 03 13:45:17.837387 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:17.837223 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd_odh-ai-gateway-infra(bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" Nov 03 13:45:17.838802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:17.838779 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" event={"ID":"ac5c3559-aa52-468d-9553-01a2b6f2498b","Type":"ContainerStarted","Data":"ce618995d422f4652bd49866bc3f29b465acba20c610f1f4008b49f56a5c4bf1"} Nov 03 13:45:17.995603 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:17.995528 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp"] Nov 03 13:45:18.147425 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.147387 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:18.263997 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.263913 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eafde52-d648-44db-a902-9394caf28cff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eafde52-d648-44db-a902-9394caf28cff-maas-api-service-ca\") pod \"0eafde52-d648-44db-a902-9394caf28cff\" (UID: \"0eafde52-d648-44db-a902-9394caf28cff\") " Nov 03 13:45:18.263997 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.263966 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eafde52-d648-44db-a902-9394caf28cff-kube-api-access-xhgtj\" (UniqueName: \"kubernetes.io/projected/0eafde52-d648-44db-a902-9394caf28cff-kube-api-access-xhgtj\") pod \"0eafde52-d648-44db-a902-9394caf28cff\" (UID: \"0eafde52-d648-44db-a902-9394caf28cff\") " Nov 03 13:45:18.264415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.264392 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eafde52-d648-44db-a902-9394caf28cff-maas-api-service-ca" pod "0eafde52-d648-44db-a902-9394caf28cff" (UID: "0eafde52-d648-44db-a902-9394caf28cff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:18.266174 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.266146 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eafde52-d648-44db-a902-9394caf28cff-kube-api-access-xhgtj" pod "0eafde52-d648-44db-a902-9394caf28cff" (UID: "0eafde52-d648-44db-a902-9394caf28cff"). InnerVolumeSpecName "kube-api-access-xhgtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:18.366203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.366163 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eafde52-d648-44db-a902-9394caf28cff-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.366203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.366201 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhgtj\" (UniqueName: \"kubernetes.io/projected/0eafde52-d648-44db-a902-9394caf28cff-kube-api-access-xhgtj\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.560012 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:18.559929 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:45:18.847151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.847041 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" event={"ID":"0eafde52-d648-44db-a902-9394caf28cff","Type":"ContainerDied","Data":"dd332e0bb4dcb45a6690160e39338ebd39c9bc260d4ed85a1d4965654aafefd7"} Nov 03 13:45:18.847151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.847147 4924 scope.go:122] "RemoveContainer" containerID="e9ed25d5a880d844048a9301a53d200ec589bc5aade398b8dd41b554a29714ed" Nov 03 13:45:18.847584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.847055 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp" Nov 03 13:45:18.853922 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.853896 4924 generic.go:356] "Generic (PLEG): container finished" podID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerID="ce618995d422f4652bd49866bc3f29b465acba20c610f1f4008b49f56a5c4bf1" exitCode=60 Nov 03 13:45:18.854030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.853961 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" event={"ID":"ac5c3559-aa52-468d-9553-01a2b6f2498b","Type":"ContainerDied","Data":"ce618995d422f4652bd49866bc3f29b465acba20c610f1f4008b49f56a5c4bf1"} Nov 03 13:45:18.854454 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.854369 4924 scope.go:122] "RemoveContainer" containerID="ce618995d422f4652bd49866bc3f29b465acba20c610f1f4008b49f56a5c4bf1" Nov 03 13:45:18.854661 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:18.854641 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j_odh-ai-gateway-infra(ac5c3559-aa52-468d-9553-01a2b6f2498b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" Nov 03 13:45:18.859888 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.859867 4924 scope.go:122] "RemoveContainer" containerID="8f5627fd315d072f78c0205dff691d0e41f4f60ff25072fdf925dcd1f2fa987c" Nov 03 13:45:18.972993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:18.972953 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd"] Nov 03 13:45:19.019972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.019944 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp"] Nov 03 13:45:19.024786 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.024747 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895225-cs5zp"] Nov 03 13:45:19.100375 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.100291 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j"] Nov 03 13:45:19.137671 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.137647 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:19.177509 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.177480 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-maas-api-service-ca\") pod \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\" (UID: \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\") " Nov 03 13:45:19.177680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.177610 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-kube-api-access-j2cg7\" (UniqueName: \"kubernetes.io/projected/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-kube-api-access-j2cg7\") pod \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\" (UID: \"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91\") " Nov 03 13:45:19.177881 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.177857 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-maas-api-service-ca" pod "bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" (UID: "bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:19.179699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.179676 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-kube-api-access-j2cg7" pod "bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" (UID: "bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91"). InnerVolumeSpecName "kube-api-access-j2cg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:19.279672 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.279635 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2cg7\" (UniqueName: \"kubernetes.io/projected/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-kube-api-access-j2cg7\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.279672 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.279673 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.865802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.865771 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" event={"ID":"bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91","Type":"ContainerDied","Data":"52568a05de0a971203b2430ccb0c0b4daa45e3e2030ab3bd7a7a4baef46f8e65"} Nov 03 13:45:19.865802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.865795 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd" Nov 03 13:45:19.866260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:19.865841 4924 scope.go:122] "RemoveContainer" containerID="66c8a983e5bae7f7b44ba081c11badb26ed6c0da56aa5e90bb6d4c8f2c09e5b9" Nov 03 13:45:20.019161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.019133 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:20.091633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.091596 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5c3559-aa52-468d-9553-01a2b6f2498b-kube-api-access-wpghp\" (UniqueName: \"kubernetes.io/projected/ac5c3559-aa52-468d-9553-01a2b6f2498b-kube-api-access-wpghp\") pod \"ac5c3559-aa52-468d-9553-01a2b6f2498b\" (UID: \"ac5c3559-aa52-468d-9553-01a2b6f2498b\") " Nov 03 13:45:20.091825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.091651 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5c3559-aa52-468d-9553-01a2b6f2498b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c3559-aa52-468d-9553-01a2b6f2498b-maas-api-service-ca\") pod \"ac5c3559-aa52-468d-9553-01a2b6f2498b\" (UID: \"ac5c3559-aa52-468d-9553-01a2b6f2498b\") " Nov 03 13:45:20.092117 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.092061 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5c3559-aa52-468d-9553-01a2b6f2498b-maas-api-service-ca" pod "ac5c3559-aa52-468d-9553-01a2b6f2498b" (UID: "ac5c3559-aa52-468d-9553-01a2b6f2498b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:20.094032 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.094007 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5c3559-aa52-468d-9553-01a2b6f2498b-kube-api-access-wpghp" pod "ac5c3559-aa52-468d-9553-01a2b6f2498b" (UID: "ac5c3559-aa52-468d-9553-01a2b6f2498b"). InnerVolumeSpecName "kube-api-access-wpghp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:20.103612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.103584 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd"] Nov 03 13:45:20.109912 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.109887 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8d8af2-29895225-wk7jd"] Nov 03 13:45:20.192946 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.192915 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpghp\" (UniqueName: \"kubernetes.io/projected/ac5c3559-aa52-468d-9553-01a2b6f2498b-kube-api-access-wpghp\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:20.192946 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.192940 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c3559-aa52-468d-9553-01a2b6f2498b-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:20.712050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.711932 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:20.866951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.866907 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:20.878781 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.878753 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" Nov 03 13:45:20.878952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.878777 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j" event={"ID":"ac5c3559-aa52-468d-9553-01a2b6f2498b","Type":"ContainerDied","Data":"2c6157c17f58b4f5f683a2327ee4f609fba255e3dfed61b35f4ccc8da47617f1"} Nov 03 13:45:20.878952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.878850 4924 scope.go:122] "RemoveContainer" containerID="ce618995d422f4652bd49866bc3f29b465acba20c610f1f4008b49f56a5c4bf1" Nov 03 13:45:20.951007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.950972 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eafde52-d648-44db-a902-9394caf28cff" path="/var/lib/kubelet/pods/0eafde52-d648-44db-a902-9394caf28cff/volumes" Nov 03 13:45:20.951527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:20.951504 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" path="/var/lib/kubelet/pods/bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91/volumes" Nov 03 13:45:21.038624 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:21.038536 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j"] Nov 03 13:45:21.042842 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:21.042814 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895225-tqr7j"] Nov 03 13:45:21.510804 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:21.510770 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:22.953057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:22.953014 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" path="/var/lib/kubelet/pods/ac5c3559-aa52-468d-9553-01a2b6f2498b/volumes" Nov 03 13:45:25.462246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.462201 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq"] Nov 03 13:45:25.462754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.462463 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" podUID="93637a67-e509-46c1-8951-eae64693d920" containerName="maas-api" containerID="cri-o://6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3" gracePeriod=30 Nov 03 13:45:25.588430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.588400 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:25.761434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.761403 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:45:25.865491 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.865452 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93637a67-e509-46c1-8951-eae64693d920-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93637a67-e509-46c1-8951-eae64693d920-openshift-service-ca\") pod \"93637a67-e509-46c1-8951-eae64693d920\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " Nov 03 13:45:25.865718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.865551 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-maas-api-tls\") pod \"93637a67-e509-46c1-8951-eae64693d920\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " Nov 03 13:45:25.865718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.865646 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-metrics-certs\") pod \"93637a67-e509-46c1-8951-eae64693d920\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " Nov 03 13:45:25.865718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.865710 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93637a67-e509-46c1-8951-eae64693d920-kube-api-access-7hr28\" (UniqueName: \"kubernetes.io/projected/93637a67-e509-46c1-8951-eae64693d920-kube-api-access-7hr28\") pod \"93637a67-e509-46c1-8951-eae64693d920\" (UID: \"93637a67-e509-46c1-8951-eae64693d920\") " Nov 03 13:45:25.866699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.866651 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93637a67-e509-46c1-8951-eae64693d920-openshift-service-ca" pod "93637a67-e509-46c1-8951-eae64693d920" (UID: "93637a67-e509-46c1-8951-eae64693d920"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:25.893335 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.888899 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-maas-api-tls" pod "93637a67-e509-46c1-8951-eae64693d920" (UID: "93637a67-e509-46c1-8951-eae64693d920"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:25.897768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.897623 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-metrics-certs" pod "93637a67-e509-46c1-8951-eae64693d920" (UID: "93637a67-e509-46c1-8951-eae64693d920"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:25.897768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.897739 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93637a67-e509-46c1-8951-eae64693d920-kube-api-access-7hr28" pod "93637a67-e509-46c1-8951-eae64693d920" (UID: "93637a67-e509-46c1-8951-eae64693d920"). InnerVolumeSpecName "kube-api-access-7hr28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:25.914904 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.914867 4924 generic.go:356] "Generic (PLEG): container finished" podID="93637a67-e509-46c1-8951-eae64693d920" containerID="6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3" exitCode=0 Nov 03 13:45:25.915079 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.914917 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" Nov 03 13:45:25.915079 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.914913 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" event={"ID":"93637a67-e509-46c1-8951-eae64693d920","Type":"ContainerDied","Data":"6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3"} Nov 03 13:45:25.915175 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.915062 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq" event={"ID":"93637a67-e509-46c1-8951-eae64693d920","Type":"ContainerDied","Data":"738ff28fb631941e991b1e3f11b552f896b2cee9d13e17867f2d11a58250e388"} Nov 03 13:45:25.915175 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.915131 4924 scope.go:122] "RemoveContainer" containerID="6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3" Nov 03 13:45:25.927184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.927081 4924 scope.go:122] "RemoveContainer" containerID="6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3" Nov 03 13:45:25.927398 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:25.927376 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3\": container with ID starting with 6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3 not found: ID does not exist" containerID="6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3" Nov 03 13:45:25.927466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.927408 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3"} err="failed to get container status \"6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3\": rpc error: code = NotFound desc = could not find container \"6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3\": container with ID starting with 6683650a60be7ba7190da70566938d6819f379c9e4f9babc8c915fc4604150f3 not found: ID does not exist" Nov 03 13:45:25.967322 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.967283 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93637a67-e509-46c1-8951-eae64693d920-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.967322 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.967322 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.967574 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.967336 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93637a67-e509-46c1-8951-eae64693d920-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.967574 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:25.967346 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hr28\" (UniqueName: \"kubernetes.io/projected/93637a67-e509-46c1-8951-eae64693d920-kube-api-access-7hr28\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:26.076953 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:26.076873 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq"] Nov 03 13:45:26.081782 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:26.081753 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8d8af2-5f9bff585d-xg2vq"] Nov 03 13:45:26.951877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:26.951843 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93637a67-e509-46c1-8951-eae64693d920" path="/var/lib/kubelet/pods/93637a67-e509-46c1-8951-eae64693d920/volumes" Nov 03 13:45:27.389364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:27.389291 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:27.626622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:27.626583 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:30.447759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:30.447719 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:32.329553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:32.329510 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:32.657345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:32.657313 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:33.051134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:33.051030 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:34.037126 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:34.037087 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:34.378265 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:34.378176 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:34.587248 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:34.587216 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:35.408980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:35.408939 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:40.618659 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:40.618571 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:41.974679 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:41.974620 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:43.333633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:43.333596 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:43.598609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:43.598491 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:46.365491 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:46.365457 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:48.560810 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:45:48.560773 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:45:49.033834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:49.033790 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:52.228521 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:52.228491 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:45:57.513054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:45:57.513016 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:02.795743 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:02.795709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:07.801356 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:07.801320 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:07.968254 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:07.968223 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:08.097426 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:08.097316 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:13.154717 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:13.154679 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:17.599019 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:17.598978 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc"] Nov 03 13:46:17.606092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:17.600209 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" podUID="ae7846be-e8bc-4306-aad2-ba4c17850dda" containerName="maas-api" containerID="cri-o://312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e" gracePeriod=30 Nov 03 13:46:17.664051 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:17.664008 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:17.933608 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:17.933572 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:46:18.084148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.084111 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-metrics-certs\") pod \"ae7846be-e8bc-4306-aad2-ba4c17850dda\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " Nov 03 13:46:18.084342 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.084229 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7846be-e8bc-4306-aad2-ba4c17850dda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7846be-e8bc-4306-aad2-ba4c17850dda-openshift-service-ca\") pod \"ae7846be-e8bc-4306-aad2-ba4c17850dda\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " Nov 03 13:46:18.084418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.084339 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls\") pod \"ae7846be-e8bc-4306-aad2-ba4c17850dda\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " Nov 03 13:46:18.084418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.084407 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7846be-e8bc-4306-aad2-ba4c17850dda-kube-api-access-xqc2p\" (UniqueName: \"kubernetes.io/projected/ae7846be-e8bc-4306-aad2-ba4c17850dda-kube-api-access-xqc2p\") pod \"ae7846be-e8bc-4306-aad2-ba4c17850dda\" (UID: \"ae7846be-e8bc-4306-aad2-ba4c17850dda\") " Nov 03 13:46:18.084805 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.084777 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7846be-e8bc-4306-aad2-ba4c17850dda-openshift-service-ca" pod "ae7846be-e8bc-4306-aad2-ba4c17850dda" (UID: "ae7846be-e8bc-4306-aad2-ba4c17850dda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:18.088251 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.088224 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7846be-e8bc-4306-aad2-ba4c17850dda-kube-api-access-xqc2p" pod "ae7846be-e8bc-4306-aad2-ba4c17850dda" (UID: "ae7846be-e8bc-4306-aad2-ba4c17850dda"). InnerVolumeSpecName "kube-api-access-xqc2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:18.088365 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.088224 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls" pod "ae7846be-e8bc-4306-aad2-ba4c17850dda" (UID: "ae7846be-e8bc-4306-aad2-ba4c17850dda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:18.092732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.092704 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-metrics-certs" pod "ae7846be-e8bc-4306-aad2-ba4c17850dda" (UID: "ae7846be-e8bc-4306-aad2-ba4c17850dda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:18.185928 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.185853 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.185928 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.185882 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7846be-e8bc-4306-aad2-ba4c17850dda-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.185928 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.185892 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae7846be-e8bc-4306-aad2-ba4c17850dda-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.185928 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.185901 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqc2p\" (UniqueName: \"kubernetes.io/projected/ae7846be-e8bc-4306-aad2-ba4c17850dda-kube-api-access-xqc2p\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.196626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.196590 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:18.291947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.291908 4924 generic.go:356] "Generic (PLEG): container finished" podID="ae7846be-e8bc-4306-aad2-ba4c17850dda" containerID="312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e" exitCode=0 Nov 03 13:46:18.292147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.291963 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" Nov 03 13:46:18.292147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.291998 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" event={"ID":"ae7846be-e8bc-4306-aad2-ba4c17850dda","Type":"ContainerDied","Data":"312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e"} Nov 03 13:46:18.292147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.292116 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc" event={"ID":"ae7846be-e8bc-4306-aad2-ba4c17850dda","Type":"ContainerDied","Data":"a08e68cfb6ca6184f53e885ddca488ba7a66282c081eb9c32f9866da6c9289aa"} Nov 03 13:46:18.292351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.292172 4924 scope.go:122] "RemoveContainer" containerID="312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e" Nov 03 13:46:18.302298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.302279 4924 scope.go:122] "RemoveContainer" containerID="312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e" Nov 03 13:46:18.302611 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:46:18.302594 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e\": container with ID starting with 312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e not found: ID does not exist" containerID="312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e" Nov 03 13:46:18.302661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.302619 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e"} err="failed to get container status \"312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e\": rpc error: code = NotFound desc = could not find container \"312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e\": container with ID starting with 312682ad64ef5d1e02c6e889243dbb7ec3605c28cf82066c53d8f9be6249983e not found: ID does not exist" Nov 03 13:46:18.309927 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.309901 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:18.449736 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.449651 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc"] Nov 03 13:46:18.454266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.454234 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a770b9-689d55856c-j6bnc"] Nov 03 13:46:18.559963 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:46:18.559928 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:46:18.950545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:18.950505 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7846be-e8bc-4306-aad2-ba4c17850dda" path="/var/lib/kubelet/pods/ae7846be-e8bc-4306-aad2-ba4c17850dda/volumes" Nov 03 13:46:19.614882 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:19.614844 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:19.737938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:19.737896 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:20.180477 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:20.180433 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:20.196761 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:20.196720 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn"] Nov 03 13:46:20.197100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:20.197038 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" containerID="cri-o://ca06fe0e2a61a05d6d6a65cd0f361efa9df535d7e71f6be515370c2b8cfc98aa" gracePeriod=30 Nov 03 13:46:20.578648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:20.578552 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:20.991722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:20.991676 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:23.965846 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.965807 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm"] Nov 03 13:46:23.966552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966533 4924 cpu_manager.go:414] "containerMap: removing container" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:46:23.966618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966555 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:46:23.966618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966575 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:46:23.966618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966585 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:46:23.966618 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966616 4924 cpu_manager.go:414] "containerMap: removing container" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966623 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966640 4924 cpu_manager.go:414] "containerMap: removing container" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966647 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966671 4924 cpu_manager.go:414] "containerMap: removing container" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966679 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966698 4924 cpu_manager.go:414] "containerMap: removing container" podUID="93637a67-e509-46c1-8951-eae64693d920" containerName="maas-api" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966707 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93637a67-e509-46c1-8951-eae64693d920" containerName="maas-api" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966717 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966724 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966743 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ae7846be-e8bc-4306-aad2-ba4c17850dda" containerName="maas-api" Nov 03 13:46:23.966766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966751 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7846be-e8bc-4306-aad2-ba4c17850dda" containerName="maas-api" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966877 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966890 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966900 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966909 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966918 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966926 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="93637a67-e509-46c1-8951-eae64693d920" containerName="maas-api" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966934 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966944 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7846be-e8bc-4306-aad2-ba4c17850dda" containerName="maas-api" Nov 03 13:46:23.967157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.966952 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:46:23.970566 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:23.970545 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.041367 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.041322 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm"] Nov 03 13:46:24.041956 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.041923 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:24.064279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.064056 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4z2\" (UniqueName: \"kubernetes.io/projected/82697fc4-111c-4ff3-bd19-1630ddd86f43-kube-api-access-mh4z2\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.064279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.064151 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-dshm\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.064279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.064204 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-model-cache\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.064279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.064248 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-home\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.064605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.064548 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-tmp-dir\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.064675 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.064611 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82697fc4-111c-4ff3-bd19-1630ddd86f43-tls-certs\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.066476 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.066451 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-4b9658\"/\"default-dockercfg-jc8f8\"" Nov 03 13:46:24.066620 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.066473 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-4b9658\"/\"openshift-service-ca.crt\"" Nov 03 13:46:24.066620 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.066530 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-4b9658\"/\"praxis-route-4b9658-kserve-self-signed-certs\"" Nov 03 13:46:24.066620 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.066559 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-4b9658\"/\"kube-root-ca.crt\"" Nov 03 13:46:24.165959 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.165915 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4z2\" (UniqueName: \"kubernetes.io/projected/82697fc4-111c-4ff3-bd19-1630ddd86f43-kube-api-access-mh4z2\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.166200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.165974 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-dshm\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.166200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.166024 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-model-cache\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.166200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.166061 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-home\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.166200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.166180 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-tmp-dir\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.166470 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.166218 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82697fc4-111c-4ff3-bd19-1630ddd86f43-tls-certs\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.166640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.166589 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-model-cache\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.166768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.166653 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-home\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.166768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.166689 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-tmp-dir\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.169633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.169325 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-dshm\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.169633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.169435 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82697fc4-111c-4ff3-bd19-1630ddd86f43-tls-certs\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.223655 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.223587 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4z2\" (UniqueName: \"kubernetes.io/projected/82697fc4-111c-4ff3-bd19-1630ddd86f43-kube-api-access-mh4z2\") pod \"praxis-route-4b9658-kserve-69b5449c49-wmbqm\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.283498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.283450 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:24.673863 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:24.673834 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm"] Nov 03 13:46:24.674577 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:46:24.674546 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82697fc4_111c_4ff3_bd19_1630ddd86f43.slice/crio-dd46c3776449da383e12b97cd2ae72199469d34ff24e60247bf36efb1f1bef93 WatchSource:0}: Error finding container dd46c3776449da383e12b97cd2ae72199469d34ff24e60247bf36efb1f1bef93: Status 404 returned error can't find the container with id dd46c3776449da383e12b97cd2ae72199469d34ff24e60247bf36efb1f1bef93 Nov 03 13:46:25.202508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.202462 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:25.341746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.341702 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" event={"ID":"82697fc4-111c-4ff3-bd19-1630ddd86f43","Type":"ContainerStarted","Data":"39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491"} Nov 03 13:46:25.341948 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.341760 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" event={"ID":"82697fc4-111c-4ff3-bd19-1630ddd86f43","Type":"ContainerStarted","Data":"dd46c3776449da383e12b97cd2ae72199469d34ff24e60247bf36efb1f1bef93"} Nov 03 13:46:25.344235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.344205 4924 generic.go:356] "Generic (PLEG): container finished" podID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerID="ca06fe0e2a61a05d6d6a65cd0f361efa9df535d7e71f6be515370c2b8cfc98aa" exitCode=0 Nov 03 13:46:25.344377 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.344281 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" event={"ID":"a6a5ca16-634c-4334-a8fd-3263acaee476","Type":"ContainerDied","Data":"ca06fe0e2a61a05d6d6a65cd0f361efa9df535d7e71f6be515370c2b8cfc98aa"} Nov 03 13:46:25.344452 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.344379 4924 scope.go:122] "RemoveContainer" containerID="d7f45b16403a7fbf28b73d161a7670ea06c2c854d37d01a9f6be729fde3304aa" Nov 03 13:46:25.450347 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.450287 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" podStartSLOduration=2.450273223 podStartE2EDuration="2.450273223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:25.449463662 +0000 UTC m=+2518.643867257" watchObservedRunningTime="2026-11-03 13:46:25.450273223 +0000 UTC m=+2518.644676825" Nov 03 13:46:25.485290 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.485263 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:46:25.583961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.583925 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-envoy\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584017 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-kube-api-access-n6rx9\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-kube-api-access-n6rx9\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584043 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-credential-socket\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584121 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-podinfo\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584151 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-token\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-token\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584186 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-socket\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584228 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a5ca16-634c-4334-a8fd-3263acaee476-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a5ca16-634c-4334-a8fd-3263acaee476-istiod-ca-cert\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584287 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-data\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584370 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-certs\") pod \"a6a5ca16-634c-4334-a8fd-3263acaee476\" (UID: \"a6a5ca16-634c-4334-a8fd-3263acaee476\") " Nov 03 13:46:25.584607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584429 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-socket" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:25.584607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584478 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-credential-socket" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:25.584713 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584663 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-data" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:25.584773 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584735 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a5ca16-634c-4334-a8fd-3263acaee476-istiod-ca-cert" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:25.584862 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584840 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-certs" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:25.584862 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584852 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:25.584970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584875 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a5ca16-634c-4334-a8fd-3263acaee476-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:25.584970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584889 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:25.584970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.584903 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:25.586353 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.586326 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-envoy" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:25.586825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.586804 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-podinfo" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:46:25.586995 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.586970 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-token" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:25.587091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.587049 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-kube-api-access-n6rx9" pod "a6a5ca16-634c-4334-a8fd-3263acaee476" (UID: "a6a5ca16-634c-4334-a8fd-3263acaee476"). InnerVolumeSpecName "kube-api-access-n6rx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:25.689094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.687805 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:25.689094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.687848 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:25.689094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.687861 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:25.689094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.687873 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a5ca16-634c-4334-a8fd-3263acaee476-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:25.689094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:25.687885 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6rx9\" (UniqueName: \"kubernetes.io/projected/a6a5ca16-634c-4334-a8fd-3263acaee476-kube-api-access-n6rx9\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.283811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:26.283773 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:26.353559 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:26.353532 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" Nov 03 13:46:26.353754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:26.353582 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn" event={"ID":"a6a5ca16-634c-4334-a8fd-3263acaee476","Type":"ContainerDied","Data":"e6e680d89dbed0d8ac90538ad7409e17788fe6b09add6e978c169f1ec0a4e6ba"} Nov 03 13:46:26.353754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:26.353650 4924 scope.go:122] "RemoveContainer" containerID="ca06fe0e2a61a05d6d6a65cd0f361efa9df535d7e71f6be515370c2b8cfc98aa" Nov 03 13:46:26.511943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:26.511910 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn"] Nov 03 13:46:26.560456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:26.560365 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a770b9-openshift-default-9c94d7bbc-kxcqn"] Nov 03 13:46:26.862028 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:26.861938 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:26.950838 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:26.950804 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" path="/var/lib/kubelet/pods/a6a5ca16-634c-4334-a8fd-3263acaee476/volumes" Nov 03 13:46:27.239807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:27.239774 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:27.396877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:27.396837 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:27.552187 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:27.546049 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:30.769258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:30.769209 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:30.932774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:30.932722 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:34.284600 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:34.284563 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:34.300254 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:34.300222 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:34.419231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:34.419195 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:46:35.662849 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:35.662805 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:36.200252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:36.200220 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:36.330187 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:36.330141 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:36.805898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:36.805811 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf"] Nov 03 13:46:36.806369 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:36.806056 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" podUID="d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" containerName="maas-api" containerID="cri-o://ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8" gracePeriod=30 Nov 03 13:46:36.901332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:36.901297 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:37.123306 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.123277 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:46:37.156520 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.156487 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-kube-api-access-49pg6\" (UniqueName: \"kubernetes.io/projected/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-kube-api-access-49pg6\") pod \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " Nov 03 13:46:37.156725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.156540 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-openshift-service-ca\") pod \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " Nov 03 13:46:37.156725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.156601 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-metrics-certs\") pod \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " Nov 03 13:46:37.156725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.156693 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls\") pod \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\" (UID: \"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8\") " Nov 03 13:46:37.157020 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.156990 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-openshift-service-ca" pod "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" (UID: "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:37.157220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.157202 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:37.159503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.159474 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-metrics-certs" pod "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" (UID: "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:37.159503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.159491 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-kube-api-access-49pg6" pod "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" (UID: "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8"). InnerVolumeSpecName "kube-api-access-49pg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:37.160690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.160663 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls" pod "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" (UID: "d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:37.258654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.258616 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49pg6\" (UniqueName: \"kubernetes.io/projected/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-kube-api-access-49pg6\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:37.258654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.258649 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:37.258907 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.258663 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:37.431747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.431714 4924 generic.go:356] "Generic (PLEG): container finished" podID="d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" containerID="ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8" exitCode=0 Nov 03 13:46:37.431950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.431763 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" Nov 03 13:46:37.431950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.431800 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" event={"ID":"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8","Type":"ContainerDied","Data":"ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8"} Nov 03 13:46:37.431950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.431858 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf" event={"ID":"d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8","Type":"ContainerDied","Data":"206d53e091518f3346b0e890d9701da196f49f2591735481eb41eaf9563ce69f"} Nov 03 13:46:37.431950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.431886 4924 scope.go:122] "RemoveContainer" containerID="ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8" Nov 03 13:46:37.433899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.433873 4924 generic.go:356] "Generic (PLEG): container finished" podID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerID="f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9" exitCode=0 Nov 03 13:46:37.434029 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.433918 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" event={"ID":"5266ad13-5565-49e7-8aed-c6d9ed2ea610","Type":"ContainerDied","Data":"f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9"} Nov 03 13:46:37.434420 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.434401 4924 scope.go:122] "RemoveContainer" containerID="f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9" Nov 03 13:46:37.435991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.435953 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:46:37.436106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.436048 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:46:37.436184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.436126 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:46:37.446816 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.446794 4924 scope.go:122] "RemoveContainer" containerID="ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8" Nov 03 13:46:37.447092 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:46:37.447060 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8\": container with ID starting with ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8 not found: ID does not exist" containerID="ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8" Nov 03 13:46:37.447147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.447103 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8"} err="failed to get container status \"ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8\": rpc error: code = NotFound desc = could not find container \"ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8\": container with ID starting with ca3354ef1caf29d8e473c923a926a8708b3feebf3e0f564efd49dbbc70860df8 not found: ID does not exist" Nov 03 13:46:37.591031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.590998 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf"] Nov 03 13:46:37.596261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:37.596233 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-84e22c-69455b865d-rtbrf"] Nov 03 13:46:38.470095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:38.469868 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" event={"ID":"5266ad13-5565-49e7-8aed-c6d9ed2ea610","Type":"ContainerStarted","Data":"e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1"} Nov 03 13:46:38.558094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:38.557426 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:46:38.573097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:38.562145 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:46:38.573097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:38.562221 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:46:38.768796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:38.768694 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:38.958006 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:38.957973 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" path="/var/lib/kubelet/pods/d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8/volumes" Nov 03 13:46:39.068289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:39.068189 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:39.209170 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:39.209134 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:39.555471 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:39.555432 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:46:39.555912 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:39.555512 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:46:40.555260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:40.555215 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:46:40.555440 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:40.555294 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:46:40.616900 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:40.616871 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:46:40.752596 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:40.752557 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:41.331675 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:41.331573 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:41.555601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:41.555559 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:46:41.555926 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:41.555876 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:46:42.140635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:42.140600 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:42.258582 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:42.258529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:42.555566 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:42.555534 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:46:42.555755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:42.555595 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:46:43.554707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:43.554668 4924 patch_prober.go:29] interesting pod/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:46:43.555239 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:43.554749 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:46:43.682819 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:43.682783 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:44.253971 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:44.253935 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:44.421749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:44.421701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:44.559856 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:44.559773 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:46:44.561135 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:44.561114 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:46:45.873389 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:45.873354 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:46.559284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:46.558628 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:46.740041 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:46.739753 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:48.558542 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:46:48.558509 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:46:49.968940 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:49.968906 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:51.575571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:51.575535 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:51.746977 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:51.746941 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:51.978114 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:51.978062 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:55.549705 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:55.549669 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:55.673951 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:55.673834 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:46:55.794735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:46:55.794693 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:00.874615 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:00.874573 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:05.970241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:05.970201 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:10.628595 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:10.628509 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:11.378855 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:11.378821 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:11.844043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:11.844013 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:12.483120 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:12.483084 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:12.638541 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:12.638506 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:16.620379 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:16.620343 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:16.783175 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:16.783135 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:16.980793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:16.980760 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:17.130536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:17.130492 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:18.565043 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:47:18.565003 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:47:20.579017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:20.578983 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:20.766802 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:20.766764 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:22.153977 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:22.153948 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:27.376583 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:27.376518 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:27.692352 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:27.692317 4924 scope.go:122] "RemoveContainer" containerID="eb32d7cbc1f6c3023d0cf0bb6c90ec1efbd16c4f26f4fb7f3966cfc21d25f5d8" Nov 03 13:47:31.099040 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:31.099003 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:31.236791 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:31.236747 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:32.457586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:32.457548 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:37.480525 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:37.480485 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:37.602612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:37.602580 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:37.722493 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:37.722447 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:42.697426 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:42.697393 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:47.896831 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:47.896796 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:48.558930 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:47:48.558888 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:47:51.552710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.552675 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl"] Nov 03 13:47:51.553220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553185 4924 cpu_manager.go:414] "containerMap: removing container" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:47:51.553220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553198 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:47:51.553220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553213 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" containerName="maas-api" Nov 03 13:47:51.553220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553218 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" containerName="maas-api" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553228 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553233 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553252 4924 cpu_manager.go:414] "containerMap: removing container" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553257 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553265 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553271 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5c3559-aa52-468d-9553-01a2b6f2498b" containerName="cleanup" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553341 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a13f2a-9835-46b2-bbf5-52c7c2bee2f8" containerName="maas-api" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553350 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553356 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eafde52-d648-44db-a902-9394caf28cff" containerName="cleanup" Nov 03 13:47:51.553361 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553364 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd6d01a-ed64-4ae9-ab1f-8897bef9aa91" containerName="cleanup" Nov 03 13:47:51.553801 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.553370 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" Nov 03 13:47:51.559207 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.559178 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.598399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.598363 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727"] Nov 03 13:47:51.598922 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.598905 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" Nov 03 13:47:51.598995 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.598922 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a5ca16-634c-4334-a8fd-3263acaee476" containerName="istio-proxy" Nov 03 13:47:51.602612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.602590 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.647016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.646980 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl"] Nov 03 13:47:51.648895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.648865 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.649046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.648938 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b603a448-7797-4973-b59a-ccac756fa5b9-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649133 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649032 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg6m\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-kube-api-access-shg6m\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649199 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649167 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.649256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649232 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.649309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649272 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.649309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649302 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649337 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.649415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649366 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649415 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649400 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649425 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjlr\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-kube-api-access-xrjlr\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.649555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649449 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649471 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649500 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b603a448-7797-4973-b59a-ccac756fa5b9-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649555 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649540 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.649824 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649584 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.649824 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649622 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.649824 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.649657 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.661383 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.661355 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727"] Nov 03 13:47:51.751277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751234 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.751474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751283 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.751474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751317 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.751474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751356 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.751474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751424 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b603a448-7797-4973-b59a-ccac756fa5b9-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.751474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751451 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg6m\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-kube-api-access-shg6m\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.751775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751509 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.751775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751576 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.751775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751624 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.751775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751654 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.751775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751693 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.751775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751724 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.751775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751728 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.751956 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.752215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752016 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752036 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.752215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752053 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752114 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.752215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752155 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjlr\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-kube-api-access-xrjlr\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.752215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752190 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752215 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752215 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752247 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b603a448-7797-4973-b59a-ccac756fa5b9-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752250 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b603a448-7797-4973-b59a-ccac756fa5b9-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752265 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.752673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752273 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.752673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752308 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752673 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752556 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.752993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.752935 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.754336 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.754309 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.754460 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.754347 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.754545 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.754509 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.754730 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.754710 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b603a448-7797-4973-b59a-ccac756fa5b9-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.805719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.805649 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjlr\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-kube-api-access-xrjlr\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.805719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.805678 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg6m\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-kube-api-access-shg6m\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.808716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.808680 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:51.808881 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.808862 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.879436 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.879408 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:51.917686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:51.917650 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:52.299878 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.299834 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl"] Nov 03 13:47:52.300657 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:47:52.300628 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb603a448_7797_4973_b59a_ccac756fa5b9.slice/crio-f71474a04569f6c1f2e6de15644fdb317fb4d6cac6f568f778d40a2dd85af7a0 WatchSource:0}: Error finding container f71474a04569f6c1f2e6de15644fdb317fb4d6cac6f568f778d40a2dd85af7a0: Status 404 returned error can't find the container with id f71474a04569f6c1f2e6de15644fdb317fb4d6cac6f568f778d40a2dd85af7a0 Nov 03 13:47:52.303278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.303243 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.303357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.303315 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.303357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.303345 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.306264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.306242 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727"] Nov 03 13:47:52.347250 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:47:52.347218 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cb48f9_6d2e_4860_bb98_3121a43d3cb3.slice/crio-31cc96a93404f1901ad4cc69b56541fb2b86e50049d449206c749d28a6e8fcc4 WatchSource:0}: Error finding container 31cc96a93404f1901ad4cc69b56541fb2b86e50049d449206c749d28a6e8fcc4: Status 404 returned error can't find the container with id 31cc96a93404f1901ad4cc69b56541fb2b86e50049d449206c749d28a6e8fcc4 Nov 03 13:47:52.349699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.349669 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.349785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.349752 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.349831 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.349784 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.997508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.997466 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" event={"ID":"b603a448-7797-4973-b59a-ccac756fa5b9","Type":"ContainerStarted","Data":"b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e"} Nov 03 13:47:52.998014 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.997526 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" event={"ID":"b603a448-7797-4973-b59a-ccac756fa5b9","Type":"ContainerStarted","Data":"f71474a04569f6c1f2e6de15644fdb317fb4d6cac6f568f778d40a2dd85af7a0"} Nov 03 13:47:52.999113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.999087 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" event={"ID":"57cb48f9-6d2e-4860-bb98-3121a43d3cb3","Type":"ContainerStarted","Data":"2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154"} Nov 03 13:47:52.999237 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:52.999133 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" event={"ID":"57cb48f9-6d2e-4860-bb98-3121a43d3cb3","Type":"ContainerStarted","Data":"31cc96a93404f1901ad4cc69b56541fb2b86e50049d449206c749d28a6e8fcc4"} Nov 03 13:47:53.090093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:53.089662 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:53.120691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:53.120548 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podStartSLOduration=2.120520583 podStartE2EDuration="2.120520583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:53.117541861 +0000 UTC m=+2606.311945457" watchObservedRunningTime="2026-11-03 13:47:53.120520583 +0000 UTC m=+2606.314924177" Nov 03 13:47:53.886094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:53.885446 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:47:53.894267 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:53.894224 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:47:53.894632 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:53.894589 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:47:53.918055 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:53.918021 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:53.919417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:53.919392 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 13:47:53.919531 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:53.919438 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 13:47:54.883394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:54.883356 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:47:54.884006 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:54.883971 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:47:54.918375 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:54.918333 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 13:47:54.918531 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:54.918384 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 13:47:55.884115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:55.883477 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:47:55.884115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:55.883539 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:47:55.918279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:55.918244 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 13:47:55.918449 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:55.918299 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 13:47:56.882099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:56.881404 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:47:56.882099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:56.881464 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:47:56.918695 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:56.918655 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 13:47:56.919117 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:56.918721 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 13:47:57.885821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:57.880442 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:47:57.885821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:57.880505 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:47:57.918680 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:57.918643 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 13:47:57.918853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:57.918698 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 13:47:58.707280 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:58.707238 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:47:58.880747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:58.880708 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:47:58.880950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:58.880775 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:47:58.918650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:58.918615 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 13:47:58.918839 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:58.918680 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 13:47:59.948921 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:59.948887 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:59.950858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:59.950836 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:47:59.952775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:47:59.952756 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:48:00.044653 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:00.044607 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:48:00.045436 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:00.045408 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:48:00.046332 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:00.046311 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:48:00.108793 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:00.108742 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podStartSLOduration=9.108711944 podStartE2EDuration="9.108711944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:53.244431566 +0000 UTC m=+2606.438835193" watchObservedRunningTime="2026-11-03 13:48:00.108711944 +0000 UTC m=+2613.303115540" Nov 03 13:48:03.878611 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:03.878535 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:09.162760 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:09.162727 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:09.321601 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:09.321569 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:09.440141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:09.440042 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:14.555599 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:14.555556 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:18.559058 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:48:18.559012 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:48:19.808007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:19.807960 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:23.994537 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:23.994499 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:24.629439 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:24.629405 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:24.979394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:24.979346 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:25.162490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.162442 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz"] Nov 03 13:48:25.166222 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.166199 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.218094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.217135 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2bceb1-openshift-default-dockercfg-pmj9z\"" Nov 03 13:48:25.224728 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.224686 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e6e7-629c-4479-86ab-e810221ef398-istiod-ca-cert\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.225119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.225093 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-data\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.225329 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.225311 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-socket\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.225466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.225452 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-envoy\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.225631 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.225617 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-credential-socket\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.225788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.225773 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-podinfo\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.225933 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.225918 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jktr\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-kube-api-access-8jktr\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.226116 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.226096 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-certs\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.226309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.226291 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-token\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.254965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.254880 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:25.284891 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.284855 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz"] Nov 03 13:48:25.329894 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.329832 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-socket\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.329894 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.329893 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-envoy\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.330191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.329954 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-credential-socket\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.330191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.329994 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-podinfo\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.330191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.330036 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jktr\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-kube-api-access-8jktr\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.330191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.330095 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-certs\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.330191 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.330146 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-token\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.330467 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.330200 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e6e7-629c-4479-86ab-e810221ef398-istiod-ca-cert\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.330467 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.330317 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-data\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.330879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.330847 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-data\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.331171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.331141 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-socket\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.332282 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.332243 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-credential-socket\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.338414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.335673 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-envoy\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.338414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.336061 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-certs\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.338414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.337971 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e6e7-629c-4479-86ab-e810221ef398-istiod-ca-cert\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.352095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.340887 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-podinfo\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.393006 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.392969 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jktr\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-kube-api-access-8jktr\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.393571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.393546 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-token\") pod \"e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.481203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.481169 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:25.545312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.543186 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:25.911794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.911763 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz"] Nov 03 13:48:25.912141 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:48:25.912116 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa8e6e7_629c_4479_86ab_e810221ef398.slice/crio-a3d8ffe7c6ff1d5d77df88b6b92b1b7e4a17a364e165bf53e917898d6020d7a8 WatchSource:0}: Error finding container a3d8ffe7c6ff1d5d77df88b6b92b1b7e4a17a364e165bf53e917898d6020d7a8: Status 404 returned error can't find the container with id a3d8ffe7c6ff1d5d77df88b6b92b1b7e4a17a364e165bf53e917898d6020d7a8 Nov 03 13:48:25.914578 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.914542 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:25.914687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.914623 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:25.914687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:25.914666 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:26.315295 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:26.315190 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" event={"ID":"ffa8e6e7-629c-4479-86ab-e810221ef398","Type":"ContainerStarted","Data":"029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215"} Nov 03 13:48:26.315295 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:26.315254 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" event={"ID":"ffa8e6e7-629c-4479-86ab-e810221ef398","Type":"ContainerStarted","Data":"a3d8ffe7c6ff1d5d77df88b6b92b1b7e4a17a364e165bf53e917898d6020d7a8"} Nov 03 13:48:26.429626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:26.427424 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" podStartSLOduration=1.427404709 podStartE2EDuration="1.427404709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:26.426247932 +0000 UTC m=+2639.620651540" watchObservedRunningTime="2026-11-03 13:48:26.427404709 +0000 UTC m=+2639.621808304" Nov 03 13:48:26.482412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:26.482386 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:26.488053 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:26.488029 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:27.322972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:27.322934 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:27.323940 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:27.323921 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:48:29.277637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:29.277597 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:29.459002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:29.458961 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:29.606625 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:29.606541 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:30.644749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:30.644685 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:31.423484 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:31.423453 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:32.626377 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:32.626339 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:32.804544 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:32.804495 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:35.559890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:35.559849 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:36.279257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:36.279218 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:39.803851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:39.803759 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:40.514301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.514255 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm"] Nov 03 13:48:40.610433 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.610398 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:40.633701 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.633663 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl"] Nov 03 13:48:40.639667 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.639639 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.694630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.688925 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5faf5d-openshift-default-dockercfg-tgctm\"" Nov 03 13:48:40.695093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.695049 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl"] Nov 03 13:48:40.725174 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725133 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-credential-socket\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.725340 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725250 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-socket\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.725340 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725295 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-certs\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.725434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725345 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsts9\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-kube-api-access-wsts9\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.725482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725443 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-data\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.725482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725475 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32447f3e-2aef-482f-8c7c-08626f6698e6-istiod-ca-cert\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.725556 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725529 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-token\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.725591 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725574 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-envoy\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.725797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.725769 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-podinfo\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.738966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.738936 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:40.846153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.834494 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-credential-socket\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.859227 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-socket\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.859282 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-certs\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.859336 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsts9\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-kube-api-access-wsts9\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.859376 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-data\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.859403 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32447f3e-2aef-482f-8c7c-08626f6698e6-istiod-ca-cert\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.859472 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-token\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.859531 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-envoy\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.859697 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-podinfo\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.849112 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-credential-socket\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.860873 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-socket\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.861143 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-certs\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.861768 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32447f3e-2aef-482f-8c7c-08626f6698e6-istiod-ca-cert\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.866162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.862010 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-data\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.880254 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.880217 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-envoy\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.898514 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.898331 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-podinfo\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.915792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.915729 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsts9\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-kube-api-access-wsts9\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.917276 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.917248 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-token\") pod \"e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:40.956470 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:40.956433 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:41.013190 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:41.013118 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:41.401672 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:41.401646 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl"] Nov 03 13:48:41.403552 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:48:41.403525 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32447f3e_2aef_482f_8c7c_08626f6698e6.slice/crio-4dc071e70dcade7163d8afed0bc1fa06806182f5e6fbdc9d8c931ef0878e6a84 WatchSource:0}: Error finding container 4dc071e70dcade7163d8afed0bc1fa06806182f5e6fbdc9d8c931ef0878e6a84: Status 404 returned error can't find the container with id 4dc071e70dcade7163d8afed0bc1fa06806182f5e6fbdc9d8c931ef0878e6a84 Nov 03 13:48:41.406444 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:41.406394 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:41.406557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:41.406486 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:41.406557 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:41.406537 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:41.436805 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:41.436772 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" event={"ID":"32447f3e-2aef-482f-8c7c-08626f6698e6","Type":"ContainerStarted","Data":"4dc071e70dcade7163d8afed0bc1fa06806182f5e6fbdc9d8c931ef0878e6a84"} Nov 03 13:48:41.582829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:41.582797 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:42.444257 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:42.444218 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" event={"ID":"32447f3e-2aef-482f-8c7c-08626f6698e6","Type":"ContainerStarted","Data":"820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c"} Nov 03 13:48:42.957903 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:42.957861 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:42.966588 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:42.966556 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:43.080417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:43.080360 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" podStartSLOduration=3.080344964 podStartE2EDuration="3.080344964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:42.647878089 +0000 UTC m=+2655.842281682" watchObservedRunningTime="2026-11-03 13:48:43.080344964 +0000 UTC m=+2656.274748558" Nov 03 13:48:43.451687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:43.451656 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:43.452632 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:43.452604 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:48:43.847005 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:43.846902 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:45.054289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:45.054255 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:45.691048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:45.691010 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:45.902528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:45.902489 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:46.073124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:46.073013 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:46.228421 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:46.228387 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:46.416091 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:46.416035 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:47.326890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:47.326853 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:48.559765 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:48:48.559721 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:48:49.037854 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:49.037818 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:49.209202 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:49.209151 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:50.868714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:50.868678 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:51.050738 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:51.050701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:51.202875 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:51.202832 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:51.929046 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:51.929005 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r"] Nov 03 13:48:51.933938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:51.933914 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.015404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.015364 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.015614 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.015423 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.015614 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.015483 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.015614 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.015522 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbhv\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-kube-api-access-bsbhv\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.015821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.015630 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.015821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.015688 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.016031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.016004 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091866a5-2c07-44ff-8816-168f6f4b7254-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.016279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.016255 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.016435 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.016411 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091866a5-2c07-44ff-8816-168f6f4b7254-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.032092 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.031225 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r"] Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.121904 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.121971 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.122002 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091866a5-2c07-44ff-8816-168f6f4b7254-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.122035 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.122101 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091866a5-2c07-44ff-8816-168f6f4b7254-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.122218 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.122252 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.122306 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.122742 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.122924 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.123228 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.123884 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091866a5-2c07-44ff-8816-168f6f4b7254-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.123930 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbhv\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-kube-api-access-bsbhv\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.126794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.124484 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.127702 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.126989 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.132703 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.132662 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091866a5-2c07-44ff-8816-168f6f4b7254-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.180210 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.176237 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.180210 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.177132 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbhv\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-kube-api-access-bsbhv\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.249993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.249953 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:52.673044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.673009 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r"] Nov 03 13:48:52.673189 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:48:52.673167 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091866a5_2c07_44ff_8816_168f6f4b7254.slice/crio-c7de2e6f1a3734f02f2c2c79ae2b5ab8059f63d152c1f0cf6e9a48cbd876af19 WatchSource:0}: Error finding container c7de2e6f1a3734f02f2c2c79ae2b5ab8059f63d152c1f0cf6e9a48cbd876af19: Status 404 returned error can't find the container with id c7de2e6f1a3734f02f2c2c79ae2b5ab8059f63d152c1f0cf6e9a48cbd876af19 Nov 03 13:48:52.675809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.675775 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:52.675917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.675849 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:52.675917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.675892 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:52.976368 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:52.976279 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:53.522582 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:53.522545 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" event={"ID":"091866a5-2c07-44ff-8816-168f6f4b7254","Type":"ContainerStarted","Data":"bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90"} Nov 03 13:48:53.522759 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:53.522631 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" event={"ID":"091866a5-2c07-44ff-8816-168f6f4b7254","Type":"ContainerStarted","Data":"c7de2e6f1a3734f02f2c2c79ae2b5ab8059f63d152c1f0cf6e9a48cbd876af19"} Nov 03 13:48:53.680037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:53.679975 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" podStartSLOduration=2.679960509 podStartE2EDuration="2.679960509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:53.677667886 +0000 UTC m=+2666.872071484" watchObservedRunningTime="2026-11-03 13:48:53.679960509 +0000 UTC m=+2666.874364102" Nov 03 13:48:54.190712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:54.188095 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:54.252094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:54.251821 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:48:54.252094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:54.251976 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 13:48:54.252094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:54.252025 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 13:48:54.393924 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:54.393882 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:55.251094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.250460 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 13:48:55.251094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.250548 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 13:48:55.544159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.542511 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" podUID="82697fc4-111c-4ff3-bd19-1630ddd86f43" containerName="main" containerID="cri-o://39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491" gracePeriod=45 Nov 03 13:48:55.880109 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.880054 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:48:55.910832 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.910806 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-tmp-dir\") pod \"82697fc4-111c-4ff3-bd19-1630ddd86f43\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " Nov 03 13:48:55.911004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.910842 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82697fc4-111c-4ff3-bd19-1630ddd86f43-tls-certs\" (UniqueName: \"kubernetes.io/secret/82697fc4-111c-4ff3-bd19-1630ddd86f43-tls-certs\") pod \"82697fc4-111c-4ff3-bd19-1630ddd86f43\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " Nov 03 13:48:55.911004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.910939 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82697fc4-111c-4ff3-bd19-1630ddd86f43-kube-api-access-mh4z2\" (UniqueName: \"kubernetes.io/projected/82697fc4-111c-4ff3-bd19-1630ddd86f43-kube-api-access-mh4z2\") pod \"82697fc4-111c-4ff3-bd19-1630ddd86f43\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " Nov 03 13:48:55.911004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.910955 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-home\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-home\") pod \"82697fc4-111c-4ff3-bd19-1630ddd86f43\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " Nov 03 13:48:55.911004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.910981 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-dshm\") pod \"82697fc4-111c-4ff3-bd19-1630ddd86f43\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " Nov 03 13:48:55.911243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.911122 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-model-cache\") pod \"82697fc4-111c-4ff3-bd19-1630ddd86f43\" (UID: \"82697fc4-111c-4ff3-bd19-1630ddd86f43\") " Nov 03 13:48:55.911243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.911130 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-tmp-dir" pod "82697fc4-111c-4ff3-bd19-1630ddd86f43" (UID: "82697fc4-111c-4ff3-bd19-1630ddd86f43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:48:55.911408 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.911383 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-model-cache" pod "82697fc4-111c-4ff3-bd19-1630ddd86f43" (UID: "82697fc4-111c-4ff3-bd19-1630ddd86f43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:48:55.911481 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.911458 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-home" pod "82697fc4-111c-4ff3-bd19-1630ddd86f43" (UID: "82697fc4-111c-4ff3-bd19-1630ddd86f43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:48:55.911729 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.911706 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:55.911860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.911733 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:55.911860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.911748 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:55.913496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.913441 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82697fc4-111c-4ff3-bd19-1630ddd86f43-tls-certs" pod "82697fc4-111c-4ff3-bd19-1630ddd86f43" (UID: "82697fc4-111c-4ff3-bd19-1630ddd86f43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:48:55.913587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.913491 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82697fc4-111c-4ff3-bd19-1630ddd86f43-kube-api-access-mh4z2" pod "82697fc4-111c-4ff3-bd19-1630ddd86f43" (UID: "82697fc4-111c-4ff3-bd19-1630ddd86f43"). InnerVolumeSpecName "kube-api-access-mh4z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:48:55.913587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:55.913465 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-dshm" pod "82697fc4-111c-4ff3-bd19-1630ddd86f43" (UID: "82697fc4-111c-4ff3-bd19-1630ddd86f43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:48:56.013100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.013050 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82697fc4-111c-4ff3-bd19-1630ddd86f43-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:56.013341 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.013315 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh4z2\" (UniqueName: \"kubernetes.io/projected/82697fc4-111c-4ff3-bd19-1630ddd86f43-kube-api-access-mh4z2\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:56.013474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.013342 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82697fc4-111c-4ff3-bd19-1630ddd86f43-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:56.251121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.251048 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 13:48:56.251638 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.251563 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 13:48:56.580144 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.580041 4924 generic.go:356] "Generic (PLEG): container finished" podID="82697fc4-111c-4ff3-bd19-1630ddd86f43" containerID="39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491" exitCode=0 Nov 03 13:48:56.580144 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.580101 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" Nov 03 13:48:56.580144 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.580111 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" event={"ID":"82697fc4-111c-4ff3-bd19-1630ddd86f43","Type":"ContainerDied","Data":"39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491"} Nov 03 13:48:56.580406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.580162 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm" event={"ID":"82697fc4-111c-4ff3-bd19-1630ddd86f43","Type":"ContainerDied","Data":"dd46c3776449da383e12b97cd2ae72199469d34ff24e60247bf36efb1f1bef93"} Nov 03 13:48:56.580406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.580180 4924 scope.go:122] "RemoveContainer" containerID="39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491" Nov 03 13:48:56.591002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.590985 4924 scope.go:122] "RemoveContainer" containerID="39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491" Nov 03 13:48:56.591380 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:48:56.591357 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491\": container with ID starting with 39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491 not found: ID does not exist" containerID="39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491" Nov 03 13:48:56.591464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.591388 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491"} err="failed to get container status \"39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491\": rpc error: code = NotFound desc = could not find container \"39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491\": container with ID starting with 39a75987ccd69320b633072996861effeaa9ef1b37fc67c7e4c558115b2f3491 not found: ID does not exist" Nov 03 13:48:56.754094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.753144 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm"] Nov 03 13:48:56.772460 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.772413 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-4b9658/praxis-route-4b9658-kserve-69b5449c49-wmbqm"] Nov 03 13:48:56.960996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:56.960962 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82697fc4-111c-4ff3-bd19-1630ddd86f43" path="/var/lib/kubelet/pods/82697fc4-111c-4ff3-bd19-1630ddd86f43/volumes" Nov 03 13:48:57.253153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:57.252506 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 13:48:57.253153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:57.252568 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 13:48:58.251536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:58.251496 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 13:48:58.251840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:58.251804 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 13:48:58.484211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:58.484174 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:48:59.250349 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:59.250311 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 13:48:59.250548 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:48:59.250380 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 13:49:00.255770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:00.255734 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:49:00.256370 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:00.256129 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:49:00.256855 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:00.256832 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:49:03.554822 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:03.554780 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:07.049834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.049797 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7"] Nov 03 13:49:07.050431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.050411 4924 cpu_manager.go:414] "containerMap: removing container" podUID="82697fc4-111c-4ff3-bd19-1630ddd86f43" containerName="main" Nov 03 13:49:07.050503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.050433 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82697fc4-111c-4ff3-bd19-1630ddd86f43" containerName="main" Nov 03 13:49:07.050563 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.050552 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="82697fc4-111c-4ff3-bd19-1630ddd86f43" containerName="main" Nov 03 13:49:07.055689 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.055670 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.156747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.156702 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7"] Nov 03 13:49:07.164877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.164834 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.165124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.164939 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.165124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.164995 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307e307c-89cf-451f-b7b2-a672e79a0f6a-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.165124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.165101 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.165333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.165162 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.165333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.165202 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.165333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.165257 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrlhk\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-kube-api-access-wrlhk\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.165333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.165286 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.165333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.165311 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.266807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.266769 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.266836 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrlhk\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-kube-api-access-wrlhk\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.266864 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267002 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.266882 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267197 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267064 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267197 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267176 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267270 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267245 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307e307c-89cf-451f-b7b2-a672e79a0f6a-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267307 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267282 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267348 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267366 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267371 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267761 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267736 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.267967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.267858 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.268121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.268029 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307e307c-89cf-451f-b7b2-a672e79a0f6a-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.269568 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.269549 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.270202 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.270179 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.320830 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.320756 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.320965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.320886 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrlhk\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-kube-api-access-wrlhk\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.372108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.372003 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:07.791615 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.791579 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7"] Nov 03 13:49:07.828035 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:49:07.828004 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307e307c_89cf_451f_b7b2_a672e79a0f6a.slice/crio-432dd9c38eb6dde42f580274228604144a8d44c71193a109c24800d801b63281 WatchSource:0}: Error finding container 432dd9c38eb6dde42f580274228604144a8d44c71193a109c24800d801b63281: Status 404 returned error can't find the container with id 432dd9c38eb6dde42f580274228604144a8d44c71193a109c24800d801b63281 Nov 03 13:49:07.830613 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.830579 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:49:07.830721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.830647 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:49:07.830776 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:07.830718 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:49:08.605967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:08.605906 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:08.672910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:08.672874 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" event={"ID":"307e307c-89cf-451f-b7b2-a672e79a0f6a","Type":"ContainerStarted","Data":"7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf"} Nov 03 13:49:08.673100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:08.672960 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" event={"ID":"307e307c-89cf-451f-b7b2-a672e79a0f6a","Type":"ContainerStarted","Data":"432dd9c38eb6dde42f580274228604144a8d44c71193a109c24800d801b63281"} Nov 03 13:49:09.372292 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:09.372242 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:09.374163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:09.374135 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 13:49:09.374277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:09.374188 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 13:49:10.372605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:10.372568 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 13:49:10.373003 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:10.372638 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 13:49:11.372999 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:11.372960 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 13:49:11.373423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:11.373016 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 13:49:12.372648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:12.372612 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 13:49:12.372841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:12.372668 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 13:49:13.372633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:13.372600 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 13:49:13.373031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:13.372651 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 13:49:13.696434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:13.696399 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:14.372966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:14.372933 4924 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 13:49:14.373405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:14.372986 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 13:49:15.378019 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:15.377988 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:15.378496 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:15.378312 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:15.379134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:15.379113 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:49:15.537716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:15.537658 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" podStartSLOduration=8.537644094000001 podStartE2EDuration="8.537644094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:08.847415219 +0000 UTC m=+2682.041818818" watchObservedRunningTime="2026-11-03 13:49:15.537644094 +0000 UTC m=+2688.732047691" Nov 03 13:49:17.191690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:17.191649 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:18.218697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:18.218665 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:18.343255 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:18.343208 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:18.458634 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:18.458598 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:18.560636 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:49:18.560545 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:49:19.316100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:19.316027 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:19.440404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:19.440309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:22.746280 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:22.746244 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:23.934121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:23.934066 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:24.768814 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:24.768771 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:27.379833 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:27.379803 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:49:27.383518 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:27.383492 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:49:28.475612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:28.475574 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:28.595571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:28.595537 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:28.793542 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:28.793438 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:29.600652 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.600607 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:29.791895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.791857 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz"] Nov 03 13:49:29.797643 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.797611 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.871527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.871435 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2bceb1-serving-cert\"" Nov 03 13:49:29.875776 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.875738 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7954-e696-4dcf-90f9-ee949cd35c27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.875966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.875820 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mn2q\" (UniqueName: \"kubernetes.io/projected/a2ff7954-e696-4dcf-90f9-ee949cd35c27-kube-api-access-4mn2q\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.875966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.875866 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.875966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.875952 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.921325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.921292 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz"] Nov 03 13:49:29.977017 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.976980 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mn2q\" (UniqueName: \"kubernetes.io/projected/a2ff7954-e696-4dcf-90f9-ee949cd35c27-kube-api-access-4mn2q\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.977266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.977040 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.977266 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.977165 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.977391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.977289 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7954-e696-4dcf-90f9-ee949cd35c27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.977391 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:49:29.977305 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-serving-cert: secret "maas-api-e2e-shared-a-w2-2bceb1-serving-cert" not found Nov 03 13:49:29.977391 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:49:29.977367 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls podName:a2ff7954-e696-4dcf-90f9-ee949cd35c27 nodeName:}" failed. No retries permitted until 2026-11-03 13:49:30.477352058 +0000 UTC m=+2703.671755630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls") pod "maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" (UID: "a2ff7954-e696-4dcf-90f9-ee949cd35c27") : secret "maas-api-e2e-shared-a-w2-2bceb1-serving-cert" not found Nov 03 13:49:29.977994 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.977969 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7954-e696-4dcf-90f9-ee949cd35c27-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:29.979540 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:29.979524 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:30.031414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:30.031383 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mn2q\" (UniqueName: \"kubernetes.io/projected/a2ff7954-e696-4dcf-90f9-ee949cd35c27-kube-api-access-4mn2q\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:30.036896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:30.036830 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:30.182501 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:30.182460 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:30.298312 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:30.298275 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:30.483733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:30.483650 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:30.486128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:30.486063 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:30.715177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:30.715135 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:31.094612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:31.094585 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz"] Nov 03 13:49:31.096206 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:49:31.096174 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ff7954_e696_4dcf_90f9_ee949cd35c27.slice/crio-bdbf593666f4846279c60fcc410359e4ea43aad12aecbc4e5ce320b4254b4866 WatchSource:0}: Error finding container bdbf593666f4846279c60fcc410359e4ea43aad12aecbc4e5ce320b4254b4866: Status 404 returned error can't find the container with id bdbf593666f4846279c60fcc410359e4ea43aad12aecbc4e5ce320b4254b4866 Nov 03 13:49:31.097807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:31.097783 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:49:31.752480 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:31.752438 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:31.890165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:31.890128 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" event={"ID":"a2ff7954-e696-4dcf-90f9-ee949cd35c27","Type":"ContainerStarted","Data":"bdbf593666f4846279c60fcc410359e4ea43aad12aecbc4e5ce320b4254b4866"} Nov 03 13:49:32.897708 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:32.897672 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" event={"ID":"a2ff7954-e696-4dcf-90f9-ee949cd35c27","Type":"ContainerStarted","Data":"1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a"} Nov 03 13:49:32.898142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:32.897771 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:33.005938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:33.005883 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" podStartSLOduration=3.06516205 podStartE2EDuration="4.00586909s" totalImagesPullingTime="940.70704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:33.004698412 +0000 UTC m=+2706.199101984" watchObservedRunningTime="2026-11-03 13:49:33.00586909 +0000 UTC m=+2706.200272711" Nov 03 13:49:34.524567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:34.524536 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:35.164322 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:35.164283 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:37.076023 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:37.075987 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:38.909648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:38.909620 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 13:49:42.367364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:42.367319 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:42.603871 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:42.603835 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:44.197162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:44.197129 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:44.321835 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:44.321800 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:45.689794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:45.689757 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:46.217287 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:46.217248 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:47.052700 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:47.052664 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:48.057726 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:48.057685 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:48.260168 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:48.260108 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:48.384263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:48.384165 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:48.558162 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:49:48.558123 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:49:52.979298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:52.979266 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:53.106499 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:53.106460 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:53.246138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:53.242065 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:53.979602 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:53.979567 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:58.549819 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:58.549778 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:59.385731 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:59.385696 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:59.533792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:59.533751 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:49:59.652029 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:49:59.651926 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:04.384349 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:04.384309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:05.583267 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.583220 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65"] Nov 03 13:50:05.583795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.583587 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" podUID="e209e423-2c61-428e-beae-bbc476dc2e7e" containerName="maas-api" containerID="cri-o://a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7" gracePeriod=30 Nov 03 13:50:05.670767 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.670727 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:05.749975 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.749933 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" podUID="e209e423-2c61-428e-beae-bbc476dc2e7e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.112:8443/health\": dial tcp 10.132.0.112:8443: connect: connection refused" Nov 03 13:50:05.915853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.915819 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:50:05.983122 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.983048 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-metrics-certs\") pod \"e209e423-2c61-428e-beae-bbc476dc2e7e\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " Nov 03 13:50:05.983313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.983153 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e209e423-2c61-428e-beae-bbc476dc2e7e-kube-api-access-qlslm\" (UniqueName: \"kubernetes.io/projected/e209e423-2c61-428e-beae-bbc476dc2e7e-kube-api-access-qlslm\") pod \"e209e423-2c61-428e-beae-bbc476dc2e7e\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " Nov 03 13:50:05.983313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.983191 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e209e423-2c61-428e-beae-bbc476dc2e7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e209e423-2c61-428e-beae-bbc476dc2e7e-openshift-service-ca\") pod \"e209e423-2c61-428e-beae-bbc476dc2e7e\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " Nov 03 13:50:05.983313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.983259 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls\") pod \"e209e423-2c61-428e-beae-bbc476dc2e7e\" (UID: \"e209e423-2c61-428e-beae-bbc476dc2e7e\") " Nov 03 13:50:05.983866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.983837 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e209e423-2c61-428e-beae-bbc476dc2e7e-openshift-service-ca" pod "e209e423-2c61-428e-beae-bbc476dc2e7e" (UID: "e209e423-2c61-428e-beae-bbc476dc2e7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:05.987260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.987223 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-metrics-certs" pod "e209e423-2c61-428e-beae-bbc476dc2e7e" (UID: "e209e423-2c61-428e-beae-bbc476dc2e7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:05.989796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:05.989757 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e209e423-2c61-428e-beae-bbc476dc2e7e-kube-api-access-qlslm" pod "e209e423-2c61-428e-beae-bbc476dc2e7e" (UID: "e209e423-2c61-428e-beae-bbc476dc2e7e"). InnerVolumeSpecName "kube-api-access-qlslm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:06.005296 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.004300 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls" pod "e209e423-2c61-428e-beae-bbc476dc2e7e" (UID: "e209e423-2c61-428e-beae-bbc476dc2e7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:06.086812 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.086777 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:06.087052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.087038 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e209e423-2c61-428e-beae-bbc476dc2e7e-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:06.087174 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.087161 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlslm\" (UniqueName: \"kubernetes.io/projected/e209e423-2c61-428e-beae-bbc476dc2e7e-kube-api-access-qlslm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:06.087272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.087261 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e209e423-2c61-428e-beae-bbc476dc2e7e-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:06.114681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.114650 4924 generic.go:356] "Generic (PLEG): container finished" podID="e209e423-2c61-428e-beae-bbc476dc2e7e" containerID="a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7" exitCode=0 Nov 03 13:50:06.114880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.114697 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" Nov 03 13:50:06.114880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.114734 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" event={"ID":"e209e423-2c61-428e-beae-bbc476dc2e7e","Type":"ContainerDied","Data":"a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7"} Nov 03 13:50:06.114880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.114791 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65" event={"ID":"e209e423-2c61-428e-beae-bbc476dc2e7e","Type":"ContainerDied","Data":"811776ed291b39cf87616ff74ce8f9622c19c63f8efcbb5c6d23bbe0228051d4"} Nov 03 13:50:06.114880 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.114814 4924 scope.go:122] "RemoveContainer" containerID="a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7" Nov 03 13:50:06.126513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.126490 4924 scope.go:122] "RemoveContainer" containerID="a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7" Nov 03 13:50:06.126904 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:50:06.126870 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7\": container with ID starting with a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7 not found: ID does not exist" containerID="a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7" Nov 03 13:50:06.126962 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.126916 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7"} err="failed to get container status \"a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7\": rpc error: code = NotFound desc = could not find container \"a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7\": container with ID starting with a00b8150420e95b0f83859cdeab0435b14b2cdd44decd730428354da6c7434b7 not found: ID does not exist" Nov 03 13:50:06.290780 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.290738 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65"] Nov 03 13:50:06.304313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.304277 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ba6fbd-775cc55945-l4b65"] Nov 03 13:50:06.470444 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.470412 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:06.608737 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.608661 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:06.955515 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:06.955482 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e209e423-2c61-428e-beae-bbc476dc2e7e" path="/var/lib/kubelet/pods/e209e423-2c61-428e-beae-bbc476dc2e7e/volumes" Nov 03 13:50:07.608181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:07.608143 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:07.927698 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:07.927650 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:08.439748 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:08.439702 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:08.442360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:08.442332 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k"] Nov 03 13:50:08.442543 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:08.442524 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" containerID="cri-o://e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1" gracePeriod=30 Nov 03 13:50:09.240264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:09.240227 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:13.368683 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.368650 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:13.559459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.559426 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:13.686965 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.686909 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:13.725145 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.725122 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:50:13.796403 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796369 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-token\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-token\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796421 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istiod-ca-cert\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796454 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-credential-socket\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796487 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-podinfo\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796585 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796540 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-certs\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796603 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-socket\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796688 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-kube-api-access-b4dfp\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-kube-api-access-b4dfp\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796742 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-data\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796786 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-envoy\") pod \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\" (UID: \"5266ad13-5565-49e7-8aed-c6d9ed2ea610\") " Nov 03 13:50:13.796818 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796804 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-credential-socket" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:13.797104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796894 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-certs" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:13.797104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796927 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-socket" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:13.797104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.796997 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istiod-ca-cert" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:13.797258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.797160 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-data" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:13.797466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.797443 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:13.797570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.797471 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:13.797570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.797487 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:13.797570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.797501 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:13.797570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.797515 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:13.798749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.798726 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-podinfo" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:50:13.798841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.798795 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-token" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:13.798886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.798854 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-kube-api-access-b4dfp" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "kube-api-access-b4dfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:13.798959 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.798945 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-envoy" pod "5266ad13-5565-49e7-8aed-c6d9ed2ea610" (UID: "5266ad13-5565-49e7-8aed-c6d9ed2ea610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:13.899123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.899088 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4dfp\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-kube-api-access-b4dfp\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:13.899123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.899125 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:13.899324 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.899140 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:13.899324 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:13.899156 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5266ad13-5565-49e7-8aed-c6d9ed2ea610-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:14.174390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.174352 4924 generic.go:356] "Generic (PLEG): container finished" podID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerID="e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1" exitCode=0 Nov 03 13:50:14.174580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.174407 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" Nov 03 13:50:14.174580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.174430 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" event={"ID":"5266ad13-5565-49e7-8aed-c6d9ed2ea610","Type":"ContainerDied","Data":"e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1"} Nov 03 13:50:14.174580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.174509 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k" event={"ID":"5266ad13-5565-49e7-8aed-c6d9ed2ea610","Type":"ContainerDied","Data":"9605d379c81160fdec5dde4792dc60b8518826e43da0f0eea2a465c34d806263"} Nov 03 13:50:14.174580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.174541 4924 scope.go:122] "RemoveContainer" containerID="e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1" Nov 03 13:50:14.186357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.186339 4924 scope.go:122] "RemoveContainer" containerID="f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9" Nov 03 13:50:14.195309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.195291 4924 scope.go:122] "RemoveContainer" containerID="e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1" Nov 03 13:50:14.195596 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:50:14.195578 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1\": container with ID starting with e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1 not found: ID does not exist" containerID="e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1" Nov 03 13:50:14.195635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.195606 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1"} err="failed to get container status \"e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1\": rpc error: code = NotFound desc = could not find container \"e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1\": container with ID starting with e39c3765772a35b281216efb2c83bf0dbef56a146cf3d0cdb67bec384d759be1 not found: ID does not exist" Nov 03 13:50:14.195635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.195626 4924 scope.go:122] "RemoveContainer" containerID="f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9" Nov 03 13:50:14.195885 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:50:14.195865 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9\": container with ID starting with f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9 not found: ID does not exist" containerID="f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9" Nov 03 13:50:14.195923 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.195895 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9"} err="failed to get container status \"f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9\": rpc error: code = NotFound desc = could not find container \"f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9\": container with ID starting with f5099adc460635d2666f3c49790503c8673584358aa507164e2d72748dfbdde9 not found: ID does not exist" Nov 03 13:50:14.334616 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.334579 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k"] Nov 03 13:50:14.382708 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.382674 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-977f32-openshift-default-79c55d596f-m6p5k"] Nov 03 13:50:14.840907 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.840872 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:14.953627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.953587 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" path="/var/lib/kubelet/pods/5266ad13-5565-49e7-8aed-c6d9ed2ea610/volumes" Nov 03 13:50:14.978622 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:14.978560 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:15.129718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:15.129632 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:15.327373 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:15.327337 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:15.635510 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:15.635479 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:16.547648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:16.547594 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:16.665847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:16.665812 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:18.558825 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:50:18.558788 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:50:18.965420 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:18.965375 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:20.972859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:20.972829 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:24.673939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:24.673905 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:28.296648 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:28.296614 4924 generic.go:356] "Generic (PLEG): container finished" podID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerID="820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c" exitCode=0 Nov 03 13:50:28.297121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:28.296689 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" event={"ID":"32447f3e-2aef-482f-8c7c-08626f6698e6","Type":"ContainerDied","Data":"820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c"} Nov 03 13:50:28.297183 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:28.297155 4924 scope.go:122] "RemoveContainer" containerID="820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c" Nov 03 13:50:28.298767 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:28.298727 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:50:28.299010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:28.298843 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:50:28.299010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:28.298896 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:50:28.452278 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:28.452177 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:50:29.304333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:29.304293 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" event={"ID":"32447f3e-2aef-482f-8c7c-08626f6698e6","Type":"ContainerStarted","Data":"28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79"} Nov 03 13:50:29.876301 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:29.876257 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:29.957384 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:29.957351 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:50:29.975063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:29.972832 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 13:50:29.975063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:29.972915 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 13:50:30.957494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:30.957448 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 13:50:30.957922 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:30.957504 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 13:50:31.957552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:31.957513 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 13:50:31.957970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:31.957578 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 13:50:32.087412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:32.087373 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:32.267336 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:32.267234 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:32.957834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:32.957794 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 13:50:32.958374 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:32.957872 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 13:50:33.378764 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:33.378672 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:33.959499 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:33.959459 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 13:50:33.960118 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:33.960065 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 13:50:34.121094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:34.117602 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:34.268153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:34.265961 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:34.962147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:34.962117 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:50:34.962606 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:34.962426 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:50:34.963281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:34.963263 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 13:50:35.057313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:35.057277 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:38.094208 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:38.094166 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:38.287968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:38.287936 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:38.414277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:38.414239 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:40.573286 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:40.573253 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:42.089407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:42.089376 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:42.507185 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:42.507154 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:43.167564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:43.167492 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:43.449376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:43.446986 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:43.650095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:43.649583 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:43.797162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:43.791659 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:46.206755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:46.206720 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:48.384988 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:48.384939 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:48.559685 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:50:48.559651 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:50:48.638526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:48.638419 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:48.786530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:48.786496 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:49.476304 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:49.476273 4924 generic.go:356] "Generic (PLEG): container finished" podID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerID="029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215" exitCode=0 Nov 03 13:50:49.476707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:49.476360 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" event={"ID":"ffa8e6e7-629c-4479-86ab-e810221ef398","Type":"ContainerDied","Data":"029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215"} Nov 03 13:50:49.476883 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:49.476867 4924 scope.go:122] "RemoveContainer" containerID="029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215" Nov 03 13:50:49.478418 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:49.478386 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:50:49.478536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:49.478477 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:50:49.478536 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:49.478528 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:50:50.486234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:50.486192 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" event={"ID":"ffa8e6e7-629c-4479-86ab-e810221ef398","Type":"ContainerStarted","Data":"f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96"} Nov 03 13:50:51.481952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:51.481919 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:50:51.482164 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:51.482133 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 13:50:51.482228 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:51.482184 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 13:50:51.868360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:51.868280 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:52.482480 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:52.482434 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 13:50:52.482658 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:52.482501 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 13:50:53.164774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:53.164738 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:53.318897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:53.318848 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:53.484961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:53.484498 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 13:50:53.484961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:53.484565 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 13:50:54.482143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:54.482104 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 13:50:54.482650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:54.482180 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 13:50:55.482294 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:55.482250 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 13:50:55.482732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:55.482334 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 13:50:56.488945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:56.488798 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:50:56.489501 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:56.489325 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:50:56.490461 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:56.490414 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 13:50:57.084184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:57.084144 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:58.302763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:58.302710 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:58.961263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:58.961230 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:59.328913 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.328830 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g"] Nov 03 13:50:59.329446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.329344 4924 cpu_manager.go:414] "containerMap: removing container" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" Nov 03 13:50:59.329446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.329356 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" Nov 03 13:50:59.329446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.329383 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e209e423-2c61-428e-beae-bbc476dc2e7e" containerName="maas-api" Nov 03 13:50:59.329446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.329387 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e209e423-2c61-428e-beae-bbc476dc2e7e" containerName="maas-api" Nov 03 13:50:59.329740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.329473 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e209e423-2c61-428e-beae-bbc476dc2e7e" containerName="maas-api" Nov 03 13:50:59.329740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.329487 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" Nov 03 13:50:59.329740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.329496 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" Nov 03 13:50:59.332119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.332095 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.383466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.383428 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-d0231d-openshift-default-dockercfg-gqz5n\"" Nov 03 13:50:59.444094 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.441968 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:59.511532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.511494 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g"] Nov 03 13:50:59.545666 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.545626 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-data\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.545954 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.545912 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-token\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.546167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.546146 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-envoy\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.546343 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.546324 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-certs\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.546482 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.546452 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dzr\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-kube-api-access-v9dzr\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.546624 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.546606 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-socket\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.546803 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.546787 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-podinfo\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.546947 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.546928 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-credential-socket\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.547084 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.547052 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istiod-ca-cert\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.653960 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.653849 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:50:59.657107 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657082 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-podinfo\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657130 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-credential-socket\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657151 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istiod-ca-cert\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657318 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657212 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-data\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657318 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657251 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-token\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657318 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657276 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-envoy\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657325 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-certs\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657351 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dzr\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-kube-api-access-v9dzr\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657390 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-socket\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657612 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-credential-socket\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657632 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-data\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.657896 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.657869 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-certs\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.658043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.658017 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istiod-ca-cert\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.658129 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.658100 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-socket\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.659912 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.659890 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-podinfo\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.660141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.660124 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-envoy\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.713413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.713369 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-token\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.714686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.714641 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dzr\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-kube-api-access-v9dzr\") pod \"e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:50:59.945597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:50:59.945564 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:51:00.364948 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:51:00.364913 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d34a5_63e2_446b_ba23_3a1e80112dc7.slice/crio-f8c0687d44efa90bd8a18c79ae2a9953f5fe00db659e68d0eb3707229cdebb93 WatchSource:0}: Error finding container f8c0687d44efa90bd8a18c79ae2a9953f5fe00db659e68d0eb3707229cdebb93: Status 404 returned error can't find the container with id f8c0687d44efa90bd8a18c79ae2a9953f5fe00db659e68d0eb3707229cdebb93 Nov 03 13:51:00.365873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:00.365841 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g"] Nov 03 13:51:00.367766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:00.367735 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:51:00.367843 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:00.367802 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:51:00.367843 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:00.367836 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:51:00.578336 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:00.578290 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" event={"ID":"e94d34a5-63e2-446b-ba23-3a1e80112dc7","Type":"ContainerStarted","Data":"018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b"} Nov 03 13:51:00.578530 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:00.578369 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" event={"ID":"e94d34a5-63e2-446b-ba23-3a1e80112dc7","Type":"ContainerStarted","Data":"f8c0687d44efa90bd8a18c79ae2a9953f5fe00db659e68d0eb3707229cdebb93"} Nov 03 13:51:00.691449 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:00.691381 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" podStartSLOduration=1.691360756 podStartE2EDuration="1.691360756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:00.687743254 +0000 UTC m=+2793.882146848" watchObservedRunningTime="2026-11-03 13:51:00.691360756 +0000 UTC m=+2793.885764352" Nov 03 13:51:00.950848 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:00.950772 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:51:01.952064 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:01.952036 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:51:02.591349 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:02.591318 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:51:02.592527 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:02.592507 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:51:02.809716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:02.809684 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:03.441171 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:03.441124 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:03.637932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:03.637898 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:03.770498 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:03.770416 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:06.809627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:06.809587 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:06.940550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:06.940510 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:08.639723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:08.639682 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:10.203339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:10.203303 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:10.318755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:10.318718 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:13.457425 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:13.457381 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:14.030886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:14.030850 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:15.300464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:15.300422 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:16.415541 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:16.415505 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:18.558551 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:51:18.558516 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:51:20.291299 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:20.291263 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:20.421235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:20.421199 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:21.914422 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:21.914372 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:22.431033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:22.430983 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:22.707474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:22.707374 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:23.685507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:23.685456 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:23.832505 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:23.832452 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:25.923289 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:25.923254 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:28.305942 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:28.305891 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:31.580882 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:31.580834 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:32.420407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:32.420339 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:37.049182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:37.049087 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:37.217346 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:37.217306 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:37.345547 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:37.345400 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:42.074913 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:42.074874 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:44.103347 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:44.103309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:44.951814 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:44.951783 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn"] Nov 03 13:51:44.952337 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:44.952321 4924 cpu_manager.go:414] "containerMap: removing container" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" Nov 03 13:51:44.952402 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:44.952339 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5266ad13-5565-49e7-8aed-c6d9ed2ea610" containerName="istio-proxy" Nov 03 13:51:44.956063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:44.956044 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.005952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.005901 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn"] Nov 03 13:51:45.048243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.048213 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:51:45.048404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.048350 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:51:45.049887 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.049858 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.050012 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.049902 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mf5l\" (UniqueName: \"kubernetes.io/projected/b5791ee5-768a-40ed-bb1e-0816cc3b5518-kube-api-access-4mf5l\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.050088 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.050057 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.151735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.151702 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.152249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.151817 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.152249 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.151842 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mf5l\" (UniqueName: \"kubernetes.io/projected/b5791ee5-768a-40ed-bb1e-0816cc3b5518-kube-api-access-4mf5l\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.152579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.152549 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.154157 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.154135 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.206711 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.206646 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mf5l\" (UniqueName: \"kubernetes.io/projected/b5791ee5-768a-40ed-bb1e-0816cc3b5518-kube-api-access-4mf5l\") pod \"maas-api-revoke-sub-6b01fc95-bbwzn\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.268737 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.268707 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:45.640275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.640249 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn"] Nov 03 13:51:45.641268 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:51:45.641237 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5791ee5_768a_40ed_bb1e_0816cc3b5518.slice/crio-002e354c5cbf42c8e7e939ed527a67f3be846e1a563c495b364c8ce96c8f5c24 WatchSource:0}: Error finding container 002e354c5cbf42c8e7e939ed527a67f3be846e1a563c495b364c8ce96c8f5c24: Status 404 returned error can't find the container with id 002e354c5cbf42c8e7e939ed527a67f3be846e1a563c495b364c8ce96c8f5c24 Nov 03 13:51:45.887895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:45.887809 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" event={"ID":"b5791ee5-768a-40ed-bb1e-0816cc3b5518","Type":"ContainerStarted","Data":"002e354c5cbf42c8e7e939ed527a67f3be846e1a563c495b364c8ce96c8f5c24"} Nov 03 13:51:46.896008 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:46.895924 4924 generic.go:356] "Generic (PLEG): container finished" podID="b5791ee5-768a-40ed-bb1e-0816cc3b5518" containerID="6c044aa23c81f31f38c293925ddfb591bf7c1133abd8851cc31c126970bd6ce2" exitCode=0 Nov 03 13:51:46.896407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:46.896010 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" event={"ID":"b5791ee5-768a-40ed-bb1e-0816cc3b5518","Type":"ContainerDied","Data":"6c044aa23c81f31f38c293925ddfb591bf7c1133abd8851cc31c126970bd6ce2"} Nov 03 13:51:48.060404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.060376 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:48.184967 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.184936 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-openshift-ca\") pod \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " Nov 03 13:51:48.185205 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.185055 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-xks-ca\") pod \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " Nov 03 13:51:48.185205 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.185161 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5791ee5-768a-40ed-bb1e-0816cc3b5518-kube-api-access-4mf5l\" (UniqueName: \"kubernetes.io/projected/b5791ee5-768a-40ed-bb1e-0816cc3b5518-kube-api-access-4mf5l\") pod \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\" (UID: \"b5791ee5-768a-40ed-bb1e-0816cc3b5518\") " Nov 03 13:51:48.185531 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.185504 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-openshift-ca" pod "b5791ee5-768a-40ed-bb1e-0816cc3b5518" (UID: "b5791ee5-768a-40ed-bb1e-0816cc3b5518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:48.187564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.187536 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-xks-ca" pod "b5791ee5-768a-40ed-bb1e-0816cc3b5518" (UID: "b5791ee5-768a-40ed-bb1e-0816cc3b5518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:48.187991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.187968 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5791ee5-768a-40ed-bb1e-0816cc3b5518-kube-api-access-4mf5l" pod "b5791ee5-768a-40ed-bb1e-0816cc3b5518" (UID: "b5791ee5-768a-40ed-bb1e-0816cc3b5518"). InnerVolumeSpecName "kube-api-access-4mf5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:48.286982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.286941 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:48.286982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.286975 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mf5l\" (UniqueName: \"kubernetes.io/projected/b5791ee5-768a-40ed-bb1e-0816cc3b5518-kube-api-access-4mf5l\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:48.286982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.286988 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5791ee5-768a-40ed-bb1e-0816cc3b5518-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:48.558499 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:51:48.558410 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:51:48.910204 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.910177 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" Nov 03 13:51:48.910377 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.910212 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn" event={"ID":"b5791ee5-768a-40ed-bb1e-0816cc3b5518","Type":"ContainerDied","Data":"002e354c5cbf42c8e7e939ed527a67f3be846e1a563c495b364c8ce96c8f5c24"} Nov 03 13:51:48.910377 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:48.910267 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="002e354c5cbf42c8e7e939ed527a67f3be846e1a563c495b364c8ce96c8f5c24" Nov 03 13:51:49.301672 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:49.301568 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:49.444722 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:49.444681 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:49.563394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:49.563267 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:52.110710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:52.110678 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:52.238726 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:52.238684 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:53.793542 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:53.793498 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:54.327621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:54.327581 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:54.808246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:54.808199 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:57.559292 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:57.559251 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:58.488841 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:58.488802 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:58.676821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:58.676783 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:51:59.790376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:51:59.790343 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:00.456117 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:00.456050 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:02.552890 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:02.552847 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:02.744033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:02.743995 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:02.945241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:02.945198 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:05.882796 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:05.882757 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:06.005270 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:06.005222 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:06.124423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:06.124390 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:06.240308 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:06.240269 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:09.753446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:09.753407 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:09.873866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:09.873829 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:11.396424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:11.396384 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:13.435811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:13.435777 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:18.558766 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:52:18.558719 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:52:18.910036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:18.910006 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:20.757739 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:20.757701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:24.577664 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:24.577621 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:29.992355 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:29.992316 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:32.737165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:32.737125 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:35.064494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:35.064455 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:40.368765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:40.368729 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:40.728502 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:40.728461 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:45.473169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:45.473128 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:48.560303 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:52:48.560259 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:52:50.892857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:50.892803 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:51.108552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:51.108513 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:51.327023 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:51.326986 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:51.941320 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:51.941279 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:52.230778 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:52.230673 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:55.753486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:55.753453 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:55.936839 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:55.936785 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:56.064713 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:56.064596 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:52:56.686277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:52:56.686241 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:00.145773 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:00.145739 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:00.328378 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:00.328343 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:01.127907 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:01.127871 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:02.187879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:02.187838 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:06.483169 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.483133 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h"] Nov 03 13:53:06.483684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.483669 4924 cpu_manager.go:414] "containerMap: removing container" podUID="b5791ee5-768a-40ed-bb1e-0816cc3b5518" containerName="revoke-keys" Nov 03 13:53:06.483727 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.483685 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5791ee5-768a-40ed-bb1e-0816cc3b5518" containerName="revoke-keys" Nov 03 13:53:06.483772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.483763 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5791ee5-768a-40ed-bb1e-0816cc3b5518" containerName="revoke-keys" Nov 03 13:53:06.487153 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.487137 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.544404 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.544369 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tmp-dir\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.544604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.544426 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-home\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.544604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.544510 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-model-cache\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.544604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.544543 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-dshm\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.544774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.544614 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tls-certs\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.544774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.544640 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2nt\" (UniqueName: \"kubernetes.io/projected/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-kube-api-access-cr2nt\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.592258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.592215 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"kube-root-ca.crt\"" Nov 03 13:53:06.592258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.592214 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"openshift-service-ca.crt\"" Nov 03 13:53:06.592258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.592252 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"auth-test-model-2bceb1-kserve-self-signed-certs\"" Nov 03 13:53:06.592579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.592350 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"default-dockercfg-rxrxl\"" Nov 03 13:53:06.595458 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.595432 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h"] Nov 03 13:53:06.643659 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.643565 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:06.645668 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.645629 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tmp-dir\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.645840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.645710 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-home\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.645840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.645748 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-model-cache\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.645840 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.645777 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-dshm\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.646031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.645878 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tls-certs\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.646031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.645918 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2nt\" (UniqueName: \"kubernetes.io/projected/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-kube-api-access-cr2nt\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.649613 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.649582 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tmp-dir\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.649855 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.649836 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-home\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.650110 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.650066 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-model-cache\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.653106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.652406 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-dshm\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.653106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.653043 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tls-certs\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.702546 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.702513 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2nt\" (UniqueName: \"kubernetes.io/projected/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-kube-api-access-cr2nt\") pod \"auth-test-model-2bceb1-kserve-75f56dbccc-4f94h\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:06.800413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:06.800312 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:07.180279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.180250 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h"] Nov 03 13:53:07.180902 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:53:07.180878 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a7925bb_39c1_4877_8428_5fe2f7ec5a9f.slice/crio-bbbc2fea35d3009c8b16b91aabe0b7a4a284221d7954b7f28de7470faf499186 WatchSource:0}: Error finding container bbbc2fea35d3009c8b16b91aabe0b7a4a284221d7954b7f28de7470faf499186: Status 404 returned error can't find the container with id bbbc2fea35d3009c8b16b91aabe0b7a4a284221d7954b7f28de7470faf499186 Nov 03 13:53:07.341121 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.341064 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv"] Nov 03 13:53:07.345902 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.345878 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.375614 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.375584 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.375775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.375657 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311839e4-3e8a-4e9c-be23-98239fc44c41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.375775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.375714 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.375870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.375844 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6sqc\" (UniqueName: \"kubernetes.io/projected/311839e4-3e8a-4e9c-be23-98239fc44c41-kube-api-access-j6sqc\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.395283 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.395248 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-5ae63d-serving-cert\"" Nov 03 13:53:07.398566 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.398541 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv"] Nov 03 13:53:07.451768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.451682 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" event={"ID":"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f","Type":"ContainerStarted","Data":"9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b"} Nov 03 13:53:07.451768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.451724 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" event={"ID":"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f","Type":"ContainerStarted","Data":"bbbc2fea35d3009c8b16b91aabe0b7a4a284221d7954b7f28de7470faf499186"} Nov 03 13:53:07.476830 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.476789 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311839e4-3e8a-4e9c-be23-98239fc44c41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.477021 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.476881 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.477539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.477512 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6sqc\" (UniqueName: \"kubernetes.io/projected/311839e4-3e8a-4e9c-be23-98239fc44c41-kube-api-access-j6sqc\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.477713 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.477692 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.477902 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:53:07.477874 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-serving-cert: secret "maas-api-e2e-shared-a-w4-5ae63d-serving-cert" not found Nov 03 13:53:07.478464 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:53:07.477932 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls podName:311839e4-3e8a-4e9c-be23-98239fc44c41 nodeName:}" failed. No retries permitted until 2026-11-03 13:53:07.977913993 +0000 UTC m=+2921.172317564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls") pod "maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" (UID: "311839e4-3e8a-4e9c-be23-98239fc44c41") : secret "maas-api-e2e-shared-a-w4-5ae63d-serving-cert" not found Nov 03 13:53:07.483134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.478839 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311839e4-3e8a-4e9c-be23-98239fc44c41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.483134 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.480195 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.524188 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.524160 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:07.532140 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.532108 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6sqc\" (UniqueName: \"kubernetes.io/projected/311839e4-3e8a-4e9c-be23-98239fc44c41-kube-api-access-j6sqc\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.565344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.565284 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" podStartSLOduration=1.565264744 podStartE2EDuration="1.565264744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:07.561426133 +0000 UTC m=+2920.755829739" watchObservedRunningTime="2026-11-03 13:53:07.565264744 +0000 UTC m=+2920.759668339" Nov 03 13:53:07.677421 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.677381 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:07.801020 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.800931 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:07.805895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.805768 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:07.983935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.983891 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:07.986329 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:07.986304 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:08.262632 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:08.262596 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:08.631889 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:08.631856 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv"] Nov 03 13:53:08.633258 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:53:08.633227 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311839e4_3e8a_4e9c_be23_98239fc44c41.slice/crio-f42d5a8a0f322e8fe72e22f9eecb1b32a7640a9ed97662ed2dcf590af0187f37 WatchSource:0}: Error finding container f42d5a8a0f322e8fe72e22f9eecb1b32a7640a9ed97662ed2dcf590af0187f37: Status 404 returned error can't find the container with id f42d5a8a0f322e8fe72e22f9eecb1b32a7640a9ed97662ed2dcf590af0187f37 Nov 03 13:53:09.470528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:09.470486 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" event={"ID":"311839e4-3e8a-4e9c-be23-98239fc44c41","Type":"ContainerStarted","Data":"f42d5a8a0f322e8fe72e22f9eecb1b32a7640a9ed97662ed2dcf590af0187f37"} Nov 03 13:53:10.483785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:10.483731 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" event={"ID":"311839e4-3e8a-4e9c-be23-98239fc44c41","Type":"ContainerStarted","Data":"1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24"} Nov 03 13:53:10.484322 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:10.484183 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:10.601177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:10.601114 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" podStartSLOduration=2.7172220620000003 podStartE2EDuration="3.601098943s" totalImagesPullingTime="883.876881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:10.597312035 +0000 UTC m=+2923.791715629" watchObservedRunningTime="2026-11-03 13:53:10.601098943 +0000 UTC m=+2923.795502538" Nov 03 13:53:12.833112 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:12.833044 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:16.497654 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:16.497618 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 13:53:16.801062 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:16.800977 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:16.814207 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:16.814176 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:17.578710 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:17.578676 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:53:18.582230 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:53:18.582188 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:53:19.016685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:19.016649 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:19.154274 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:19.154229 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:19.283799 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:19.283678 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:20.599865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:20.599829 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:21.070050 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:21.070008 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:21.465681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:21.465643 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:24.718538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:24.718480 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:26.636232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:26.636144 4924 generic.go:356] "Generic (PLEG): container finished" podID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerID="018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b" exitCode=0 Nov 03 13:53:26.636232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:26.636218 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" event={"ID":"e94d34a5-63e2-446b-ba23-3a1e80112dc7","Type":"ContainerDied","Data":"018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b"} Nov 03 13:53:26.636685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:26.636674 4924 scope.go:122] "RemoveContainer" containerID="018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b" Nov 03 13:53:26.638490 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:26.638450 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:53:26.638565 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:26.638550 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:53:26.638617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:26.638601 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:53:27.666226 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:27.666185 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" event={"ID":"e94d34a5-63e2-446b-ba23-3a1e80112dc7","Type":"ContainerStarted","Data":"da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc"} Nov 03 13:53:27.947150 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:27.946365 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:53:27.958093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:27.948124 4924 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 13:53:27.958093 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:27.948183 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 13:53:28.950095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:28.946511 4924 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 13:53:28.950095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:28.946576 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 13:53:29.949097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:29.949007 4924 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 13:53:29.949097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:29.949086 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 13:53:29.957211 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:29.956549 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:30.951097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:30.950174 4924 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 13:53:30.951097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:30.950236 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 13:53:31.948097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:31.946400 4924 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 13:53:31.948097 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:31.946470 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 13:53:32.591847 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:32.591808 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:53:32.946785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:32.946112 4924 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 13:53:32.946785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:32.946184 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 13:53:33.100235 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:33.100202 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:33.280352 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:33.280259 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:33.950344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:33.950314 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:53:33.951375 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:33.951355 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 13:53:35.068756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:35.068720 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:40.174981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:40.174912 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:44.666808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:44.666763 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:45.217380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:45.217340 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:45.367652 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:45.367604 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:45.640705 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:45.640605 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:48.562436 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:53:48.562389 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:53:49.308811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:49.308777 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:49.487140 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:49.487059 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:49.620470 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:49.620392 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:50.550566 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:50.550520 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:52.609851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:52.609820 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:52.784219 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:52.784179 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:54.342993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:54.342957 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r"] Nov 03 13:53:54.343564 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:54.343283 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" containerID="cri-o://bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90" gracePeriod=30 Nov 03 13:53:55.821287 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:55.821252 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:53:59.606227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.606165 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:53:59.686660 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.686624 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-istio-token\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-istio-token\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.686866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.686689 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-kube-api-access-bsbhv\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-kube-api-access-bsbhv\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.686866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.686722 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-socket\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.686866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.686833 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-data\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.687039 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.686873 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091866a5-2c07-44ff-8816-168f6f4b7254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091866a5-2c07-44ff-8816-168f6f4b7254-istiod-ca-cert\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.687039 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.686985 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-credential-socket\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.687162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687029 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-socket" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:59.687162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687040 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/091866a5-2c07-44ff-8816-168f6f4b7254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091866a5-2c07-44ff-8816-168f6f4b7254-istio-podinfo\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.687162 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687153 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-data" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:59.687316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687191 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-certs\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.687316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687235 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-envoy\") pod \"091866a5-2c07-44ff-8816-168f6f4b7254\" (UID: \"091866a5-2c07-44ff-8816-168f6f4b7254\") " Nov 03 13:53:59.687423 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687387 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-credential-socket" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:59.687480 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687441 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091866a5-2c07-44ff-8816-168f6f4b7254-istiod-ca-cert" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:53:59.687635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687612 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-certs" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:59.687883 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687859 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.687945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687890 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.687945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687903 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.687945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687916 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.687945 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.687929 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091866a5-2c07-44ff-8816-168f6f4b7254-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.689550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.689524 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-kube-api-access-bsbhv" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "kube-api-access-bsbhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:59.689695 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.689576 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/091866a5-2c07-44ff-8816-168f6f4b7254-istio-podinfo" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:53:59.689695 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.689618 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-envoy" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:59.689997 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.689967 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-istio-token" pod "091866a5-2c07-44ff-8816-168f6f4b7254" (UID: "091866a5-2c07-44ff-8816-168f6f4b7254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:59.789538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.789496 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091866a5-2c07-44ff-8816-168f6f4b7254-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.789538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.789528 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091866a5-2c07-44ff-8816-168f6f4b7254-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.789538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.789537 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.789851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.789548 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbhv\" (UniqueName: \"kubernetes.io/projected/091866a5-2c07-44ff-8816-168f6f4b7254-kube-api-access-bsbhv\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:59.929330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.929297 4924 generic.go:356] "Generic (PLEG): container finished" podID="091866a5-2c07-44ff-8816-168f6f4b7254" containerID="bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90" exitCode=0 Nov 03 13:53:59.929508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.929345 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" Nov 03 13:53:59.929508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.929384 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" event={"ID":"091866a5-2c07-44ff-8816-168f6f4b7254","Type":"ContainerDied","Data":"bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90"} Nov 03 13:53:59.929508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.929444 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r" event={"ID":"091866a5-2c07-44ff-8816-168f6f4b7254","Type":"ContainerDied","Data":"c7de2e6f1a3734f02f2c2c79ae2b5ab8059f63d152c1f0cf6e9a48cbd876af19"} Nov 03 13:53:59.929508 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.929468 4924 scope.go:122] "RemoveContainer" containerID="bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90" Nov 03 13:53:59.940918 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.940898 4924 scope.go:122] "RemoveContainer" containerID="bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90" Nov 03 13:53:59.941288 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:53:59.941267 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90\": container with ID starting with bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90 not found: ID does not exist" containerID="bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90" Nov 03 13:53:59.941360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:53:59.941299 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90"} err="failed to get container status \"bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90\": rpc error: code = NotFound desc = could not find container \"bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90\": container with ID starting with bf19b791f46c1fb355a90ae30bf7352874816987d88925a64e676b32ae650a90 not found: ID does not exist" Nov 03 13:54:00.084351 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:00.084314 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r"] Nov 03 13:54:00.089140 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:00.089111 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-dp82r"] Nov 03 13:54:00.799476 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:00.799432 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:00.951476 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:00.951445 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" path="/var/lib/kubelet/pods/091866a5-2c07-44ff-8816-168f6f4b7254/volumes" Nov 03 13:54:00.977042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:00.977009 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:01.101907 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:01.101815 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:06.150520 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:06.150461 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:07.632681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:07.632638 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:09.467553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:09.467519 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7"] Nov 03 13:54:09.467994 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:09.467745 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" containerID="cri-o://7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf" gracePeriod=30 Nov 03 13:54:11.257474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:11.257413 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:14.740892 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.740836 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:54:14.789587 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.789560 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-data\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.789774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.789609 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-kube-api-access-wrlhk\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-kube-api-access-wrlhk\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.789774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.789703 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-podinfo\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.789774 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.789740 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-credential-socket\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.790010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.789777 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-socket\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.790010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.789806 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-token\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-token\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.790010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.789817 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-data" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.790010 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.789898 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-envoy\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.790243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790015 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-certs\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.790243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790016 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-credential-socket" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.790243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790038 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-socket" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.790243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790129 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307e307c-89cf-451f-b7b2-a672e79a0f6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307e307c-89cf-451f-b7b2-a672e79a0f6a-istiod-ca-cert\") pod \"307e307c-89cf-451f-b7b2-a672e79a0f6a\" (UID: \"307e307c-89cf-451f-b7b2-a672e79a0f6a\") " Nov 03 13:54:14.790464 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790272 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-certs" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.790576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790539 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307e307c-89cf-451f-b7b2-a672e79a0f6a-istiod-ca-cert" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:54:14.790718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790619 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.790718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790633 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.790718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790644 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.790718 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.790653 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.791976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.791955 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-kube-api-access-wrlhk" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "kube-api-access-wrlhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:54:14.791976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.791958 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-podinfo" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:54:14.792425 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.792402 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-envoy" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.792492 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.792412 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-token" pod "307e307c-89cf-451f-b7b2-a672e79a0f6a" (UID: "307e307c-89cf-451f-b7b2-a672e79a0f6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:54:14.892170 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.892137 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/307e307c-89cf-451f-b7b2-a672e79a0f6a-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.892170 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.892165 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrlhk\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-kube-api-access-wrlhk\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.892170 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.892175 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.892474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.892183 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.892474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:14.892194 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/307e307c-89cf-451f-b7b2-a672e79a0f6a-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:15.051835 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.051752 4924 generic.go:356] "Generic (PLEG): container finished" podID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerID="7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf" exitCode=0 Nov 03 13:54:15.051835 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.051798 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" Nov 03 13:54:15.052038 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.051829 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" event={"ID":"307e307c-89cf-451f-b7b2-a672e79a0f6a","Type":"ContainerDied","Data":"7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf"} Nov 03 13:54:15.052038 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.051882 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7" event={"ID":"307e307c-89cf-451f-b7b2-a672e79a0f6a","Type":"ContainerDied","Data":"432dd9c38eb6dde42f580274228604144a8d44c71193a109c24800d801b63281"} Nov 03 13:54:15.052038 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.051906 4924 scope.go:122] "RemoveContainer" containerID="7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf" Nov 03 13:54:15.061588 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.061570 4924 scope.go:122] "RemoveContainer" containerID="7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf" Nov 03 13:54:15.061865 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:54:15.061849 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf\": container with ID starting with 7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf not found: ID does not exist" containerID="7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf" Nov 03 13:54:15.061914 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.061877 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf"} err="failed to get container status \"7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf\": rpc error: code = NotFound desc = could not find container \"7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf\": container with ID starting with 7adc1b33c825a42d7c5a0831b521daf8894cabec97f386bcc207b906b926f1cf not found: ID does not exist" Nov 03 13:54:15.208742 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.208703 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7"] Nov 03 13:54:15.213828 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:15.213797 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-8dvc7"] Nov 03 13:54:16.295241 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:16.295203 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:16.950766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:16.950723 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" path="/var/lib/kubelet/pods/307e307c-89cf-451f-b7b2-a672e79a0f6a/volumes" Nov 03 13:54:18.562509 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:54:18.562461 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:54:21.355400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:21.355358 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:25.391952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:25.391913 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:25.799749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:25.799716 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:26.459181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:26.458656 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:26.635380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:26.629241 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:27.199605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:27.199549 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:27.435414 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:27.435384 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:54:27.441754 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:27.441728 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:54:30.656098 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:30.656053 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:30.892488 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:30.892455 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:31.048681 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:31.048641 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:32.746298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.746264 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd"] Nov 03 13:54:32.746770 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.746757 4924 cpu_manager.go:414] "containerMap: removing container" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" Nov 03 13:54:32.746816 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.746771 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" Nov 03 13:54:32.746816 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.746799 4924 cpu_manager.go:414] "containerMap: removing container" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" Nov 03 13:54:32.746816 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.746805 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" Nov 03 13:54:32.746917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.746901 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="307e307c-89cf-451f-b7b2-a672e79a0f6a" containerName="istio-proxy" Nov 03 13:54:32.746917 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.746913 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="091866a5-2c07-44ff-8816-168f6f4b7254" containerName="istio-proxy" Nov 03 13:54:32.751576 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.751558 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.804713 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.804681 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd"] Nov 03 13:54:32.821605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.821571 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:32.841037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.841001 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-dshm\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.841255 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.841188 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tmp-dir\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.841340 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.841295 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhk4r\" (UniqueName: \"kubernetes.io/projected/24b4e1c2-98cf-489d-adb3-9abd680eb85b-kube-api-access-qhk4r\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.841391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.841360 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tls-certs\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.841520 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.841499 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-model-cache\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.841558 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.841544 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-home\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.846865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.846839 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"openshift-service-ca.crt\"" Nov 03 13:54:32.847099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.847066 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"kube-root-ca.crt\"" Nov 03 13:54:32.847240 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.847225 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"default-dockercfg-hln72\"" Nov 03 13:54:32.848043 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.848025 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"auth-test-model-d341af-kserve-self-signed-certs\"" Nov 03 13:54:32.939231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.939112 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:32.946159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.945037 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-dshm\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.946159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.945134 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tmp-dir\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.946159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.945169 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhk4r\" (UniqueName: \"kubernetes.io/projected/24b4e1c2-98cf-489d-adb3-9abd680eb85b-kube-api-access-qhk4r\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.946159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.945224 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tls-certs\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.946159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.945292 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-model-cache\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.946159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.945324 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-home\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.946159 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.945812 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-home\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.947437 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.947412 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tmp-dir\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.952020 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.948180 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-model-cache\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.957426 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.956545 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-dshm\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:32.964772 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:32.964716 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tls-certs\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:33.001565 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:33.001492 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhk4r\" (UniqueName: \"kubernetes.io/projected/24b4e1c2-98cf-489d-adb3-9abd680eb85b-kube-api-access-qhk4r\") pod \"auth-test-model-d341af-kserve-56cb74fc85-n4drd\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:33.065349 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:33.065304 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:33.463966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:33.463934 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:33.465338 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:33.465310 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd"] Nov 03 13:54:33.466445 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:54:33.466419 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24b4e1c2_98cf_489d_adb3_9abd680eb85b.slice/crio-d03111257a4548f20223ae7daed7deddbedac300289d4e8b5dea1205097f1740 WatchSource:0}: Error finding container d03111257a4548f20223ae7daed7deddbedac300289d4e8b5dea1205097f1740: Status 404 returned error can't find the container with id d03111257a4548f20223ae7daed7deddbedac300289d4e8b5dea1205097f1740 Nov 03 13:54:33.951867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:33.951833 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:34.128019 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:34.127984 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:34.279829 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:34.279726 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" event={"ID":"24b4e1c2-98cf-489d-adb3-9abd680eb85b","Type":"ContainerStarted","Data":"61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e"} Nov 03 13:54:34.280115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:34.280054 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" event={"ID":"24b4e1c2-98cf-489d-adb3-9abd680eb85b","Type":"ContainerStarted","Data":"d03111257a4548f20223ae7daed7deddbedac300289d4e8b5dea1205097f1740"} Nov 03 13:54:34.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:34.391654 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" podStartSLOduration=2.39163991 podStartE2EDuration="2.39163991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:34.38956777 +0000 UTC m=+3007.583971365" watchObservedRunningTime="2026-11-03 13:54:34.39163991 +0000 UTC m=+3007.586043528" Nov 03 13:54:35.065417 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:35.065384 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:38.201909 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:38.201805 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:38.324026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:38.323987 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:43.066390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:43.066346 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:43.080615 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:43.080582 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:43.360184 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:43.360104 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:54:43.380667 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:43.380637 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:48.518469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:48.518439 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:48.572833 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:54:48.572787 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:54:53.546443 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:53.546408 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:58.509682 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:58.509636 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:58.679637 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:58.679590 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:54:58.802139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:54:58.802041 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:03.748866 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:03.748814 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:08.802733 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:08.802699 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:13.817465 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:13.817428 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:18.558109 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:55:18.558051 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:55:18.998396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:18.998347 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:20.611055 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:20.611017 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:24.029163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:24.029117 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:24.489707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:24.489674 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:29.611409 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:29.611370 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:33.805757 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:33.805711 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:34.768436 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:34.768395 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:39.803711 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:39.803673 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:40.198570 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:40.198528 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727"] Nov 03 13:55:40.198768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:40.198745 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" containerID="cri-o://2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154" gracePeriod=30 Nov 03 13:55:44.798039 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:44.798003 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:44.953616 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:44.953580 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:55:44.953794 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:44.953660 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:55:44.960950 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:44.960921 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:45.082972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.082878 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:45.558431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.558401 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:55:45.632284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632178 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-certs\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.632284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632277 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istiod-ca-cert\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.632539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632320 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-token\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-token\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.632539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632368 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-envoy\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.632539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632511 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-certs" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:55:45.632714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632680 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-socket\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.632780 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632727 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-kube-api-access-xrjlr\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-kube-api-access-xrjlr\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.632879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632849 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-data\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.633014 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.632918 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-podinfo\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.633562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.633221 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istiod-ca-cert" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:55:45.633562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.633327 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-credential-socket\") pod \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\" (UID: \"57cb48f9-6d2e-4860-bb98-3121a43d3cb3\") " Nov 03 13:55:45.633562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.633443 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-data" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:55:45.633562 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.633510 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-socket" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:55:45.633893 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.633752 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-credential-socket" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:55:45.634763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.633952 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.634763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.633980 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.634763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.633997 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.634763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.634013 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.634763 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.634029 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.636792 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.636729 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-token" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:55:45.636981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.636959 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-kube-api-access-xrjlr" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "kube-api-access-xrjlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:55:45.637455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.637429 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-podinfo" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:55:45.637577 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.637452 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-envoy" pod "57cb48f9-6d2e-4860-bb98-3121a43d3cb3" (UID: "57cb48f9-6d2e-4860-bb98-3121a43d3cb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:55:45.738339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.736048 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.738339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.736150 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.738339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.736169 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjlr\" (UniqueName: \"kubernetes.io/projected/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-kube-api-access-xrjlr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.738339 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.736202 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cb48f9-6d2e-4860-bb98-3121a43d3cb3-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:45.763690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.763630 4924 generic.go:356] "Generic (PLEG): container finished" podID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerID="2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154" exitCode=0 Nov 03 13:55:45.764173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.764095 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" event={"ID":"57cb48f9-6d2e-4860-bb98-3121a43d3cb3","Type":"ContainerDied","Data":"2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154"} Nov 03 13:55:45.764413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.764351 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" event={"ID":"57cb48f9-6d2e-4860-bb98-3121a43d3cb3","Type":"ContainerDied","Data":"31cc96a93404f1901ad4cc69b56541fb2b86e50049d449206c749d28a6e8fcc4"} Nov 03 13:55:45.768096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.764545 4924 scope.go:122] "RemoveContainer" containerID="2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154" Nov 03 13:55:45.768096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.764713 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727" Nov 03 13:55:45.790808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.790782 4924 scope.go:122] "RemoveContainer" containerID="2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154" Nov 03 13:55:45.791251 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:55:45.791217 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154\": container with ID starting with 2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154 not found: ID does not exist" containerID="2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154" Nov 03 13:55:45.791363 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.791249 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154"} err="failed to get container status \"2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154\": rpc error: code = NotFound desc = could not find container \"2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154\": container with ID starting with 2d3292b9938ee48a2fb88c33f78bd033472b6922bf804220797e6c57fe63d154 not found: ID does not exist" Nov 03 13:55:45.929723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.929522 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727"] Nov 03 13:55:45.937106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:45.937038 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-77727"] Nov 03 13:55:46.951619 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:46.951588 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" path="/var/lib/kubelet/pods/57cb48f9-6d2e-4860-bb98-3121a43d3cb3/volumes" Nov 03 13:55:48.558899 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:55:48.558851 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:55:50.066606 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:50.066566 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:51.897260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.897223 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw"] Nov 03 13:55:51.897769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.897748 4924 cpu_manager.go:414] "containerMap: removing container" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" Nov 03 13:55:51.897769 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.897765 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" Nov 03 13:55:51.897939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.897926 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cb48f9-6d2e-4860-bb98-3121a43d3cb3" containerName="istio-proxy" Nov 03 13:55:51.902744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.902725 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:51.955803 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.955771 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw"] Nov 03 13:55:51.977912 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.977879 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:55:51.997059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.997028 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"openshift-service-ca.crt\"" Nov 03 13:55:51.997059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.997034 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"kube-root-ca.crt\"" Nov 03 13:55:51.997275 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.997097 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"default-dockercfg-4n97f\"" Nov 03 13:55:51.999216 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:51.999201 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"auth-test-model-60261a-kserve-self-signed-certs\"" Nov 03 13:55:52.027807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.027779 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae1de5e-75c6-4914-a625-1eee2201cdbf-tls-certs\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.028115 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.028057 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkkm\" (UniqueName: \"kubernetes.io/projected/fae1de5e-75c6-4914-a625-1eee2201cdbf-kube-api-access-qwkkm\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.028269 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.028250 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-home\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.028344 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.028284 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-tmp-dir\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.028627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.028596 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-model-cache\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.028687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.028640 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-dshm\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.129697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.129662 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-model-cache\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.129697 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.129700 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-dshm\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.129985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.129768 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae1de5e-75c6-4914-a625-1eee2201cdbf-tls-certs\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.129985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.129841 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkkm\" (UniqueName: \"kubernetes.io/projected/fae1de5e-75c6-4914-a625-1eee2201cdbf-kube-api-access-qwkkm\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.129985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.129922 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-home\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.129985 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.129950 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-tmp-dir\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.130231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.130185 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-model-cache\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.130359 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.130340 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-tmp-dir\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.130539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.130520 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-home\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.132350 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.132326 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae1de5e-75c6-4914-a625-1eee2201cdbf-tls-certs\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.132506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.132356 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-dshm\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.183983 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.183948 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkkm\" (UniqueName: \"kubernetes.io/projected/fae1de5e-75c6-4914-a625-1eee2201cdbf-kube-api-access-qwkkm\") pod \"auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.214817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.214780 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:52.591616 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:55:52.591581 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfae1de5e_75c6_4914_a625_1eee2201cdbf.slice/crio-50eef8e74ffc2e138564092284ce53d6adfef7ba7d8af40184951bd0eaf21f0f WatchSource:0}: Error finding container 50eef8e74ffc2e138564092284ce53d6adfef7ba7d8af40184951bd0eaf21f0f: Status 404 returned error can't find the container with id 50eef8e74ffc2e138564092284ce53d6adfef7ba7d8af40184951bd0eaf21f0f Nov 03 13:55:52.591765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.591742 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw"] Nov 03 13:55:52.826810 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.826772 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" event={"ID":"fae1de5e-75c6-4914-a625-1eee2201cdbf","Type":"ContainerStarted","Data":"79101288eaf6bc77de2432b0b09f86f0906972c8d1d03c4826a36b739b0076fe"} Nov 03 13:55:52.826979 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.826820 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" event={"ID":"fae1de5e-75c6-4914-a625-1eee2201cdbf","Type":"ContainerStarted","Data":"50eef8e74ffc2e138564092284ce53d6adfef7ba7d8af40184951bd0eaf21f0f"} Nov 03 13:55:52.937843 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:52.937792 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" podStartSLOduration=1.937778048 podStartE2EDuration="1.937778048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:55:52.937181893 +0000 UTC m=+3086.131585489" watchObservedRunningTime="2026-11-03 13:55:52.937778048 +0000 UTC m=+3086.132181657" Nov 03 13:55:53.215670 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:53.215593 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:55:55.161857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:55:55.161792 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:00.253316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:00.253276 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:02.216036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:02.215998 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:56:02.228715 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:02.228687 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:56:02.905916 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:02.905878 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 13:56:05.296612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:05.296579 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:06.785746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:06.785707 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:10.357407 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:10.357370 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:15.561749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:15.561712 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:16.560746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:16.560705 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:18.559167 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:56:18.559124 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:56:20.686400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:20.686362 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:22.869376 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:22.869338 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:26.216163 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:26.216125 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:31.272113 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:31.272053 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:34.137438 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:34.137389 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:34.307313 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:34.307280 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:36.328674 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:36.328635 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:39.762475 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.762440 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv"] Nov 03 13:56:39.767609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.767580 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.811516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.811477 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-dshm\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.811696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.811536 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tls-certs\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.811696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.811678 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tmp-dir\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.811809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.811769 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-model-cache\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.811943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.811917 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g429g\" (UniqueName: \"kubernetes.io/projected/3185385c-8a2f-4e96-9eaa-59a3caeed42f-kube-api-access-g429g\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.812034 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.812016 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-home\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.830846 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.830801 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:39.863401 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.863367 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-92546f\"/\"explicit-model-92546f-kserve-self-signed-certs\"" Nov 03 13:56:39.863741 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.863719 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-92546f\"/\"default-dockercfg-gf54d\"" Nov 03 13:56:39.863906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.863873 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-92546f\"/\"openshift-service-ca.crt\"" Nov 03 13:56:39.863906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.863895 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-92546f\"/\"kube-root-ca.crt\"" Nov 03 13:56:39.866756 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.866730 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv"] Nov 03 13:56:39.922363 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.921848 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tmp-dir\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.923709 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.922665 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-model-cache\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.924714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.922315 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tmp-dir\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.924887 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.924574 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-model-cache\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.925165 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.925132 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g429g\" (UniqueName: \"kubernetes.io/projected/3185385c-8a2f-4e96-9eaa-59a3caeed42f-kube-api-access-g429g\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.925721 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.925701 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-home\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.926142 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.926124 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-home\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.926279 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.926170 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-dshm\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.926440 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.926422 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tls-certs\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.930617 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.930575 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tls-certs\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.931234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.931184 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-dshm\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:39.979203 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:39.979174 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g429g\" (UniqueName: \"kubernetes.io/projected/3185385c-8a2f-4e96-9eaa-59a3caeed42f-kube-api-access-g429g\") pod \"explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:40.083989 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:40.083906 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:40.478686 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:40.478656 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv"] Nov 03 13:56:40.479634 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:56:40.479609 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3185385c_8a2f_4e96_9eaa_59a3caeed42f.slice/crio-7c823daa55a791999b6de2ded0e99c1cec824917cf3bb0c073c1221e3ec30118 WatchSource:0}: Error finding container 7c823daa55a791999b6de2ded0e99c1cec824917cf3bb0c073c1221e3ec30118: Status 404 returned error can't find the container with id 7c823daa55a791999b6de2ded0e99c1cec824917cf3bb0c073c1221e3ec30118 Nov 03 13:56:41.208788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:41.208749 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" event={"ID":"3185385c-8a2f-4e96-9eaa-59a3caeed42f","Type":"ContainerStarted","Data":"be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da"} Nov 03 13:56:41.209322 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:41.208801 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" event={"ID":"3185385c-8a2f-4e96-9eaa-59a3caeed42f","Type":"ContainerStarted","Data":"7c823daa55a791999b6de2ded0e99c1cec824917cf3bb0c073c1221e3ec30118"} Nov 03 13:56:41.319521 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:41.319464 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" podStartSLOduration=2.319449767 podStartE2EDuration="2.319449767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:41.31920806 +0000 UTC m=+3134.513611654" watchObservedRunningTime="2026-11-03 13:56:41.319449767 +0000 UTC m=+3134.513853361" Nov 03 13:56:41.435246 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:41.435195 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:42.084422 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:42.084379 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:46.462506 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:46.462458 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:48.559736 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:56:48.559695 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:56:49.096935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:49.096900 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn"] Nov 03 13:56:49.141507 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:49.141472 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b01fc95-bbwzn"] Nov 03 13:56:50.084633 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:50.084598 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:50.097883 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:50.097858 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:50.282469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:50.282441 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:56:50.951867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:50.951837 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5791ee5-768a-40ed-bb1e-0816cc3b5518" path="/var/lib/kubelet/pods/b5791ee5-768a-40ed-bb1e-0816cc3b5518/volumes" Nov 03 13:56:51.545910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:51.545872 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:56:57.458455 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:56:57.458413 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:01.677835 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:01.677802 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:06.859577 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:06.859539 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:09.114600 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:09.114568 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:12.318582 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:12.318544 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:17.438281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:17.438243 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:18.562997 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:57:18.562959 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:57:21.073489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:21.073454 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:22.504730 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:22.504686 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:26.047817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:26.047778 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv"] Nov 03 13:57:26.139836 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:26.139804 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:27.559138 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:27.559103 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:29.507099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:29.507038 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8"] Nov 03 13:57:29.507592 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:29.507279 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" containerID="cri-o://56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1" gracePeriod=30 Nov 03 13:57:32.690310 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:32.690271 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:33.658583 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:33.658541 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:34.489048 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.489009 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:34.798175 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.798149 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:57:34.893533 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893502 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-certs\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.893749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893574 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-socket\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.893749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893615 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-envoy\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.893749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893675 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-data\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.893749 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893708 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-token\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.893976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893816 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-certs" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:34.893976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893827 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-credential-socket\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.893976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893856 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-socket" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:34.893976 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893894 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-kube-api-access-lkgfp\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-kube-api-access-lkgfp\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.894209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893994 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-data" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:34.894209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.893994 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-podinfo\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.894209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.894027 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-credential-socket" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:34.894209 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.894062 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istiod-ca-cert\") pod \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\" (UID: \"cbfb959d-44dd-4c99-8b90-888ae2d2305e\") " Nov 03 13:57:34.894609 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.894583 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istiod-ca-cert" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:57:34.894699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.894612 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:34.894699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.894635 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:34.894699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.894652 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:34.894699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.894667 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:34.897101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.896396 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-podinfo" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:57:34.897101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.896395 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-token" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:34.897101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.896501 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-envoy" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:34.897101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.896854 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-kube-api-access-lkgfp" pod "cbfb959d-44dd-4c99-8b90-888ae2d2305e" (UID: "cbfb959d-44dd-4c99-8b90-888ae2d2305e"). InnerVolumeSpecName "kube-api-access-lkgfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:34.995809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.995777 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:34.995809 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.995807 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:34.996003 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.995817 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:34.996003 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.995826 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:34.996003 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:34.995839 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgfp\" (UniqueName: \"kubernetes.io/projected/cbfb959d-44dd-4c99-8b90-888ae2d2305e-kube-api-access-lkgfp\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:35.356474 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.356435 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c"] Nov 03 13:57:35.356955 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.356942 4924 cpu_manager.go:414] "containerMap: removing container" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" Nov 03 13:57:35.357001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.356956 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" Nov 03 13:57:35.357001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.356969 4924 cpu_manager.go:414] "containerMap: removing container" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" Nov 03 13:57:35.357001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.356974 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" Nov 03 13:57:35.357143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.357092 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" Nov 03 13:57:35.357143 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.357105 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerName="istio-proxy" Nov 03 13:57:35.360723 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.360697 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.410149 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.410113 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"kube-root-ca.crt\"" Nov 03 13:57:35.410347 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.410186 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"openshift-service-ca.crt\"" Nov 03 13:57:35.410347 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.410207 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"default-dockercfg-4g28d\"" Nov 03 13:57:35.438004 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.437964 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"auth-test-model-367381-kserve-self-signed-certs\"" Nov 03 13:57:35.442626 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.442595 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:35.461398 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.461368 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c"] Nov 03 13:57:35.500360 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.500328 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbms\" (UniqueName: \"kubernetes.io/projected/6d71c46c-cf44-4230-b70a-997a0a23e989-kube-api-access-dwbms\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.500735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.500376 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d71c46c-cf44-4230-b70a-997a0a23e989-tls-certs\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.500735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.500545 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-tmp-dir\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.500735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.500589 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-model-cache\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.500735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.500700 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-dshm\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.500876 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.500739 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-home\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.592260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.592225 4924 generic.go:356] "Generic (PLEG): container finished" podID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" containerID="56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1" exitCode=0 Nov 03 13:57:35.592466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.592278 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" Nov 03 13:57:35.592466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.592294 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" event={"ID":"cbfb959d-44dd-4c99-8b90-888ae2d2305e","Type":"ContainerDied","Data":"56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1"} Nov 03 13:57:35.592466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.592336 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8" event={"ID":"cbfb959d-44dd-4c99-8b90-888ae2d2305e","Type":"ContainerDied","Data":"9bd9cff847a86acdc2b3baf2775546ca0e9a5ff287b913a2b32bd95180c8b587"} Nov 03 13:57:35.592466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.592359 4924 scope.go:122] "RemoveContainer" containerID="56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1" Nov 03 13:57:35.601977 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.601939 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d71c46c-cf44-4230-b70a-997a0a23e989-tls-certs\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.602176 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.602153 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-tmp-dir\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.602256 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.602212 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-model-cache\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.602309 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.602284 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-dshm\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.602399 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.602378 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-home\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.602456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.602435 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbms\" (UniqueName: \"kubernetes.io/projected/6d71c46c-cf44-4230-b70a-997a0a23e989-kube-api-access-dwbms\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.602579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.602556 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-tmp-dir\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.602785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.602765 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-model-cache\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.603107 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.603065 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-home\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.603938 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.603913 4924 scope.go:122] "RemoveContainer" containerID="61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea" Nov 03 13:57:35.605096 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.605053 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-dshm\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.605363 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.605343 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d71c46c-cf44-4230-b70a-997a0a23e989-tls-certs\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.631147 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.630980 4924 scope.go:122] "RemoveContainer" containerID="56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1" Nov 03 13:57:35.631411 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:57:35.631385 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1\": container with ID starting with 56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1 not found: ID does not exist" containerID="56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1" Nov 03 13:57:35.631489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.631424 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1"} err="failed to get container status \"56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1\": rpc error: code = NotFound desc = could not find container \"56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1\": container with ID starting with 56e095c87b24dc0b75caa2258df448bd255d1708a606a09f113a6d111561b0d1 not found: ID does not exist" Nov 03 13:57:35.631489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.631450 4924 scope.go:122] "RemoveContainer" containerID="61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea" Nov 03 13:57:35.631835 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:57:35.631813 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea\": container with ID starting with 61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea not found: ID does not exist" containerID="61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea" Nov 03 13:57:35.631911 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.631846 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea"} err="failed to get container status \"61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea\": rpc error: code = NotFound desc = could not find container \"61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea\": container with ID starting with 61b87a2fe8574e7460fcc7f548d9b5ab208afced3891889221569ec3ef38a5ea not found: ID does not exist" Nov 03 13:57:35.658173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.658143 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbms\" (UniqueName: \"kubernetes.io/projected/6d71c46c-cf44-4230-b70a-997a0a23e989-kube-api-access-dwbms\") pod \"auth-test-model-367381-kserve-7b7b4db645-48z9c\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.675877 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.675848 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:35.754687 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.754620 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8"] Nov 03 13:57:35.759946 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:35.759909 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-shvr8"] Nov 03 13:57:36.095486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:36.095458 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c"] Nov 03 13:57:36.097319 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:57:36.097283 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d71c46c_cf44_4230_b70a_997a0a23e989.slice/crio-e4d4c4abb76bb0e6e79ca3182b4e26be2c170839a51261daa6ccdd5880398ef0 WatchSource:0}: Error finding container e4d4c4abb76bb0e6e79ca3182b4e26be2c170839a51261daa6ccdd5880398ef0: Status 404 returned error can't find the container with id e4d4c4abb76bb0e6e79ca3182b4e26be2c170839a51261daa6ccdd5880398ef0 Nov 03 13:57:36.600001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:36.599966 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" event={"ID":"6d71c46c-cf44-4230-b70a-997a0a23e989","Type":"ContainerStarted","Data":"f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c"} Nov 03 13:57:36.600472 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:36.600012 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" event={"ID":"6d71c46c-cf44-4230-b70a-997a0a23e989","Type":"ContainerStarted","Data":"e4d4c4abb76bb0e6e79ca3182b4e26be2c170839a51261daa6ccdd5880398ef0"} Nov 03 13:57:36.676568 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:36.676531 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:36.710746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:36.710691 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" podStartSLOduration=1.710675801 podStartE2EDuration="1.710675801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:57:36.707933769 +0000 UTC m=+3189.902337379" watchObservedRunningTime="2026-11-03 13:57:36.710675801 +0000 UTC m=+3189.905079395" Nov 03 13:57:36.963960 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:36.963920 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbfb959d-44dd-4c99-8b90-888ae2d2305e" path="/var/lib/kubelet/pods/cbfb959d-44dd-4c99-8b90-888ae2d2305e/volumes" Nov 03 13:57:37.775991 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:37.775955 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:41.069202 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.069163 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" podUID="3185385c-8a2f-4e96-9eaa-59a3caeed42f" containerName="main" containerID="cri-o://be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da" gracePeriod=45 Nov 03 13:57:41.156469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.156421 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl"] Nov 03 13:57:41.156747 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.156717 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" containerID="cri-o://b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e" gracePeriod=30 Nov 03 13:57:41.373825 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.373797 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:57:41.488999 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.488967 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3185385c-8a2f-4e96-9eaa-59a3caeed42f-kube-api-access-g429g\" (UniqueName: \"kubernetes.io/projected/3185385c-8a2f-4e96-9eaa-59a3caeed42f-kube-api-access-g429g\") pod \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " Nov 03 13:57:41.489231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.489016 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-home\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-home\") pod \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " Nov 03 13:57:41.489231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.489052 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-model-cache\") pod \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " Nov 03 13:57:41.489231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.489140 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tmp-dir\") pod \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " Nov 03 13:57:41.489231 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.489201 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tls-certs\") pod \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " Nov 03 13:57:41.489465 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.489266 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-dshm\") pod \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\" (UID: \"3185385c-8a2f-4e96-9eaa-59a3caeed42f\") " Nov 03 13:57:41.490788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.490653 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-model-cache" pod "3185385c-8a2f-4e96-9eaa-59a3caeed42f" (UID: "3185385c-8a2f-4e96-9eaa-59a3caeed42f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:41.490788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.490720 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tmp-dir" pod "3185385c-8a2f-4e96-9eaa-59a3caeed42f" (UID: "3185385c-8a2f-4e96-9eaa-59a3caeed42f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:41.490788 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.490767 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-home" pod "3185385c-8a2f-4e96-9eaa-59a3caeed42f" (UID: "3185385c-8a2f-4e96-9eaa-59a3caeed42f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:41.492789 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.492765 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3185385c-8a2f-4e96-9eaa-59a3caeed42f-kube-api-access-g429g" pod "3185385c-8a2f-4e96-9eaa-59a3caeed42f" (UID: "3185385c-8a2f-4e96-9eaa-59a3caeed42f"). InnerVolumeSpecName "kube-api-access-g429g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:41.496326 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.496294 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-dshm" pod "3185385c-8a2f-4e96-9eaa-59a3caeed42f" (UID: "3185385c-8a2f-4e96-9eaa-59a3caeed42f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:41.496424 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.496316 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tls-certs" pod "3185385c-8a2f-4e96-9eaa-59a3caeed42f" (UID: "3185385c-8a2f-4e96-9eaa-59a3caeed42f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:57:41.590897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.590863 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g429g\" (UniqueName: \"kubernetes.io/projected/3185385c-8a2f-4e96-9eaa-59a3caeed42f-kube-api-access-g429g\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:41.590897 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.590894 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:41.591128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.590907 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:41.591128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.590921 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:41.591128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.590933 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3185385c-8a2f-4e96-9eaa-59a3caeed42f-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:41.591128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.590943 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3185385c-8a2f-4e96-9eaa-59a3caeed42f-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:41.634045 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.633960 4924 generic.go:356] "Generic (PLEG): container finished" podID="3185385c-8a2f-4e96-9eaa-59a3caeed42f" containerID="be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da" exitCode=0 Nov 03 13:57:41.634045 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.634010 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" Nov 03 13:57:41.634262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.634039 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" event={"ID":"3185385c-8a2f-4e96-9eaa-59a3caeed42f","Type":"ContainerDied","Data":"be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da"} Nov 03 13:57:41.634262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.634108 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" event={"ID":"3185385c-8a2f-4e96-9eaa-59a3caeed42f","Type":"ContainerDied","Data":"7c823daa55a791999b6de2ded0e99c1cec824917cf3bb0c073c1221e3ec30118"} Nov 03 13:57:41.634262 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.634128 4924 scope.go:122] "RemoveContainer" containerID="be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da" Nov 03 13:57:41.644727 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.644707 4924 scope.go:122] "RemoveContainer" containerID="be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da" Nov 03 13:57:41.644996 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:57:41.644979 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da\": container with ID starting with be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da not found: ID does not exist" containerID="be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da" Nov 03 13:57:41.645031 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.645004 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da"} err="failed to get container status \"be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da\": rpc error: code = NotFound desc = could not find container \"be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da\": container with ID starting with be0036c7e94507d71d910e57c2ec4f8092b6a310bf589ab7dbfaa0f13c1fe2da not found: ID does not exist" Nov 03 13:57:41.790478 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.790437 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv"] Nov 03 13:57:41.795460 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:41.795426 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv"] Nov 03 13:57:42.272434 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:42.272383 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-92546f/explicit-model-92546f-kserve-7b7ff7f8c5-nvxqv" podUID="3185385c-8a2f-4e96-9eaa-59a3caeed42f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:57:42.868442 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:42.868408 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:42.960556 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:42.960523 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3185385c-8a2f-4e96-9eaa-59a3caeed42f" path="/var/lib/kubelet/pods/3185385c-8a2f-4e96-9eaa-59a3caeed42f/volumes" Nov 03 13:57:45.048026 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:45.047989 4924 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:57:45.048519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:45.048057 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:57:45.676926 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:45.676888 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:45.690001 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:45.689972 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:46.423300 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.423234 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:57:46.447456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447426 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-socket\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.447612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447464 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-istio-token\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-istio-token\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.447612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447547 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-kube-api-access-shg6m\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-kube-api-access-shg6m\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.447612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447575 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-credential-socket\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.447612 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447599 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b603a448-7797-4973-b59a-ccac756fa5b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b603a448-7797-4973-b59a-ccac756fa5b9-istiod-ca-cert\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.447807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447627 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-data\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.447807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447694 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-envoy\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.447807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447733 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-socket" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.447807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447775 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b603a448-7797-4973-b59a-ccac756fa5b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b603a448-7797-4973-b59a-ccac756fa5b9-istio-podinfo\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.448021 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447836 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-certs\") pod \"b603a448-7797-4973-b59a-ccac756fa5b9\" (UID: \"b603a448-7797-4973-b59a-ccac756fa5b9\") " Nov 03 13:57:46.448021 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447877 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-credential-socket" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.448021 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.447993 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-data" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.448220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.448092 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b603a448-7797-4973-b59a-ccac756fa5b9-istiod-ca-cert" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:57:46.448220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.448183 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-certs" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.448370 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.448349 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.448446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.448378 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.448446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.448392 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b603a448-7797-4973-b59a-ccac756fa5b9-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.448446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.448406 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.448446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.448420 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.449889 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.449865 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b603a448-7797-4973-b59a-ccac756fa5b9-istio-podinfo" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:57:46.450055 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.450041 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-kube-api-access-shg6m" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "kube-api-access-shg6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:46.450193 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.450178 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-istio-token" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:46.450260 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.450248 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-envoy" pod "b603a448-7797-4973-b59a-ccac756fa5b9" (UID: "b603a448-7797-4973-b59a-ccac756fa5b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.549706 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.549669 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b603a448-7797-4973-b59a-ccac756fa5b9-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.549706 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.549707 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.549936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.549723 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shg6m\" (UniqueName: \"kubernetes.io/projected/b603a448-7797-4973-b59a-ccac756fa5b9-kube-api-access-shg6m\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.549936 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.549739 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b603a448-7797-4973-b59a-ccac756fa5b9-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.674712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.674615 4924 generic.go:356] "Generic (PLEG): container finished" podID="b603a448-7797-4973-b59a-ccac756fa5b9" containerID="b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e" exitCode=0 Nov 03 13:57:46.674712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.674661 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" Nov 03 13:57:46.674712 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.674697 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" event={"ID":"b603a448-7797-4973-b59a-ccac756fa5b9","Type":"ContainerDied","Data":"b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e"} Nov 03 13:57:46.675007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.674759 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl" event={"ID":"b603a448-7797-4973-b59a-ccac756fa5b9","Type":"ContainerDied","Data":"f71474a04569f6c1f2e6de15644fdb317fb4d6cac6f568f778d40a2dd85af7a0"} Nov 03 13:57:46.675007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.674787 4924 scope.go:122] "RemoveContainer" containerID="b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e" Nov 03 13:57:46.687330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.687304 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 13:57:46.687635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.687616 4924 scope.go:122] "RemoveContainer" containerID="b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e" Nov 03 13:57:46.687937 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:57:46.687922 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e\": container with ID starting with b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e not found: ID does not exist" containerID="b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e" Nov 03 13:57:46.687980 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.687946 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e"} err="failed to get container status \"b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e\": rpc error: code = NotFound desc = could not find container \"b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e\": container with ID starting with b15bd54c1a54a44fa52025febb5d5ac6ef7b7dc1841aa597219a43f81db99b6e not found: ID does not exist" Nov 03 13:57:46.940716 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.940628 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl"] Nov 03 13:57:46.954776 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:46.954746 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6wqfl"] Nov 03 13:57:47.979586 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:47.979555 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:48.564564 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:57:48.564523 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:57:48.951181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:48.951144 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" path="/var/lib/kubelet/pods/b603a448-7797-4973-b59a-ccac756fa5b9/volumes" Nov 03 13:57:53.069969 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.069930 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:53.488921 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.488887 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6"] Nov 03 13:57:53.489436 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.489419 4924 cpu_manager.go:414] "containerMap: removing container" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" Nov 03 13:57:53.489436 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.489436 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" Nov 03 13:57:53.489526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.489464 4924 cpu_manager.go:414] "containerMap: removing container" podUID="3185385c-8a2f-4e96-9eaa-59a3caeed42f" containerName="main" Nov 03 13:57:53.489526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.489469 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3185385c-8a2f-4e96-9eaa-59a3caeed42f" containerName="main" Nov 03 13:57:53.489589 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.489547 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b603a448-7797-4973-b59a-ccac756fa5b9" containerName="istio-proxy" Nov 03 13:57:53.489589 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.489560 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3185385c-8a2f-4e96-9eaa-59a3caeed42f" containerName="main" Nov 03 13:57:53.494539 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.494515 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.531330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.531297 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-home\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.531516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.531341 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-tmp-dir\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.531516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.531467 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cca271-b0b2-4925-a387-cd61c4c507f9-tls-certs\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.531516 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.531498 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-dshm\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.531684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.531551 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7zz\" (UniqueName: \"kubernetes.io/projected/29cca271-b0b2-4925-a387-cd61c4c507f9-kube-api-access-jl7zz\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.531684 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.531650 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-model-cache\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.556430 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.556368 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:57:53.590352 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.590314 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-5ae63d\"/\"openshift-service-ca.crt\"" Nov 03 13:57:53.590352 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.590336 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-5ae63d\"/\"kube-root-ca.crt\"" Nov 03 13:57:53.590566 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.590398 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-5ae63d\"/\"auth-test-model-5ae63d-kserve-self-signed-certs\"" Nov 03 13:57:53.590566 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.590547 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-5ae63d\"/\"default-dockercfg-68v28\"" Nov 03 13:57:53.593691 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.593669 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6"] Nov 03 13:57:53.632779 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.632742 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cca271-b0b2-4925-a387-cd61c4c507f9-tls-certs\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.632960 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.632787 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-dshm\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.632960 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.632858 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7zz\" (UniqueName: \"kubernetes.io/projected/29cca271-b0b2-4925-a387-cd61c4c507f9-kube-api-access-jl7zz\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.632960 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.632927 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-model-cache\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.633146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.632966 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-home\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.633146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.633001 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-tmp-dir\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.633373 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.633345 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-model-cache\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.633441 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.633398 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-tmp-dir\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.633598 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.633581 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-home\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.635470 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.635442 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cca271-b0b2-4925-a387-cd61c4c507f9-tls-certs\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.635621 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.635600 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-dshm\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.687107 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.687056 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7zz\" (UniqueName: \"kubernetes.io/projected/29cca271-b0b2-4925-a387-cd61c4c507f9-kube-api-access-jl7zz\") pod \"auth-test-model-5ae63d-kserve-75b4656db9-tx9j6\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:53.806915 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:53.806825 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:54.194657 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:54.194627 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6"] Nov 03 13:57:54.195590 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:57:54.195559 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29cca271_b0b2_4925_a387_cd61c4c507f9.slice/crio-1bdd789dca5d48be0c1795bb7652fe8a383d48e0f309349c2b2ff799f04bf7da WatchSource:0}: Error finding container 1bdd789dca5d48be0c1795bb7652fe8a383d48e0f309349c2b2ff799f04bf7da: Status 404 returned error can't find the container with id 1bdd789dca5d48be0c1795bb7652fe8a383d48e0f309349c2b2ff799f04bf7da Nov 03 13:57:54.731744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:54.731700 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" event={"ID":"29cca271-b0b2-4925-a387-cd61c4c507f9","Type":"ContainerStarted","Data":"a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c"} Nov 03 13:57:54.731932 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:54.731756 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" event={"ID":"29cca271-b0b2-4925-a387-cd61c4c507f9","Type":"ContainerStarted","Data":"1bdd789dca5d48be0c1795bb7652fe8a383d48e0f309349c2b2ff799f04bf7da"} Nov 03 13:57:54.807160 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:54.807118 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:57:54.840400 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:54.840344 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" podStartSLOduration=1.840328869 podStartE2EDuration="1.840328869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:57:54.83898174 +0000 UTC m=+3208.033385335" watchObservedRunningTime="2026-11-03 13:57:54.840328869 +0000 UTC m=+3208.034732463" Nov 03 13:57:58.192806 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:57:58.192758 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:03.163348 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:03.163291 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:03.381765 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:03.381687 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:03.807412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:03.807381 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:58:03.820022 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:03.819995 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:58:04.046952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:04.046911 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:04.180851 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:04.180810 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:04.806316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:04.806280 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 13:58:05.578347 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.578305 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns"] Nov 03 13:58:05.583177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.583152 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.635247 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.635214 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:58:05.635922 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.635896 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns"] Nov 03 13:58:05.643016 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.642988 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.643181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.643025 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c4d\" (UniqueName: \"kubernetes.io/projected/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-kube-api-access-r5c4d\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.643261 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.643201 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.678287 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.678257 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:58:05.743845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.743810 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.743845 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.743848 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c4d\" (UniqueName: \"kubernetes.io/projected/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-kube-api-access-r5c4d\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.744044 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.743906 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.744606 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.744581 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.746167 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.746140 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.798388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.798353 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c4d\" (UniqueName: \"kubernetes.io/projected/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-kube-api-access-r5c4d\") pod \"maas-api-revoke-sub-4f124edb-4jmns\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:05.897012 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:05.896929 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:06.266892 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:06.266860 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns"] Nov 03 13:58:06.268145 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:58:06.268118 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbe0bfc_e2aa_4520_8609_b3209c038fe1.slice/crio-2f766f066d8b84a42d48e10729f53e01f82a19b6e97623eda19806d11836c080 WatchSource:0}: Error finding container 2f766f066d8b84a42d48e10729f53e01f82a19b6e97623eda19806d11836c080: Status 404 returned error can't find the container with id 2f766f066d8b84a42d48e10729f53e01f82a19b6e97623eda19806d11836c080 Nov 03 13:58:06.821707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:06.821613 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" event={"ID":"2cbe0bfc-e2aa-4520-8609-b3209c038fe1","Type":"ContainerStarted","Data":"dc84cc2b98eccf14ca23fa455789c60d2ee48e854f7cf46b49383991bdf9267b"} Nov 03 13:58:06.822103 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:06.821717 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" event={"ID":"2cbe0bfc-e2aa-4520-8609-b3209c038fe1","Type":"ContainerStarted","Data":"2f766f066d8b84a42d48e10729f53e01f82a19b6e97623eda19806d11836c080"} Nov 03 13:58:06.932192 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:06.932133 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" podStartSLOduration=1.932117863 podStartE2EDuration="1.932117863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:58:06.929230156 +0000 UTC m=+3220.123633770" watchObservedRunningTime="2026-11-03 13:58:06.932117863 +0000 UTC m=+3220.126521456" Nov 03 13:58:07.831491 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:07.831451 4924 generic.go:356] "Generic (PLEG): container finished" podID="2cbe0bfc-e2aa-4520-8609-b3209c038fe1" containerID="dc84cc2b98eccf14ca23fa455789c60d2ee48e854f7cf46b49383991bdf9267b" exitCode=0 Nov 03 13:58:07.831914 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:07.831540 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" event={"ID":"2cbe0bfc-e2aa-4520-8609-b3209c038fe1","Type":"ContainerDied","Data":"dc84cc2b98eccf14ca23fa455789c60d2ee48e854f7cf46b49383991bdf9267b"} Nov 03 13:58:07.889894 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:07.889818 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:08.017453 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:08.017397 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:08.674006 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:08.673932 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:08.804443 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:08.804388 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:08.999277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:08.999249 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:09.082953 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.082921 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-kube-api-access-r5c4d\" (UniqueName: \"kubernetes.io/projected/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-kube-api-access-r5c4d\") pod \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " Nov 03 13:58:09.083156 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.083057 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-xks-ca\") pod \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " Nov 03 13:58:09.083156 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.083112 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-openshift-ca\") pod \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\" (UID: \"2cbe0bfc-e2aa-4520-8609-b3209c038fe1\") " Nov 03 13:58:09.083707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.083458 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-openshift-ca" pod "2cbe0bfc-e2aa-4520-8609-b3209c038fe1" (UID: "2cbe0bfc-e2aa-4520-8609-b3209c038fe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:09.083707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.083645 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:09.085229 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.085207 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-kube-api-access-r5c4d" pod "2cbe0bfc-e2aa-4520-8609-b3209c038fe1" (UID: "2cbe0bfc-e2aa-4520-8609-b3209c038fe1"). InnerVolumeSpecName "kube-api-access-r5c4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:09.085337 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.085210 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-xks-ca" pod "2cbe0bfc-e2aa-4520-8609-b3209c038fe1" (UID: "2cbe0bfc-e2aa-4520-8609-b3209c038fe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:09.184519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.184482 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:09.184519 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.184513 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c4d\" (UniqueName: \"kubernetes.io/projected/2cbe0bfc-e2aa-4520-8609-b3209c038fe1-kube-api-access-r5c4d\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:09.847398 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.847365 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" Nov 03 13:58:09.847597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.847396 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns" event={"ID":"2cbe0bfc-e2aa-4520-8609-b3209c038fe1","Type":"ContainerDied","Data":"2f766f066d8b84a42d48e10729f53e01f82a19b6e97623eda19806d11836c080"} Nov 03 13:58:09.847597 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:09.847446 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f766f066d8b84a42d48e10729f53e01f82a19b6e97623eda19806d11836c080" Nov 03 13:58:13.783331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:13.783295 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:16.349817 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.349777 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr"] Nov 03 13:58:16.350333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.350301 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2cbe0bfc-e2aa-4520-8609-b3209c038fe1" containerName="revoke-keys" Nov 03 13:58:16.350333 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.350316 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbe0bfc-e2aa-4520-8609-b3209c038fe1" containerName="revoke-keys" Nov 03 13:58:16.350447 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.350442 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbe0bfc-e2aa-4520-8609-b3209c038fe1" containerName="revoke-keys" Nov 03 13:58:16.353981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.353957 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.414234 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.414201 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr"] Nov 03 13:58:16.451173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.451138 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:58:16.451420 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.451260 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:58:16.481544 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.481513 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvd85\" (UniqueName: \"kubernetes.io/projected/47db33e4-c501-45b8-b3f2-494660c9aefa-kube-api-access-bvd85\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.481714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.481694 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.481768 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.481738 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.582807 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.582773 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvd85\" (UniqueName: \"kubernetes.io/projected/47db33e4-c501-45b8-b3f2-494660c9aefa-kube-api-access-bvd85\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.582992 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.582903 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.582992 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.582934 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.583659 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.583632 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.585222 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.585192 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.637532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.637451 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvd85\" (UniqueName: \"kubernetes.io/projected/47db33e4-c501-45b8-b3f2-494660c9aefa-kube-api-access-bvd85\") pod \"maas-api-revoke-sub-cc96c114-5q8mr\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:16.681551 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:16.681521 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:17.058024 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:58:17.057985 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47db33e4_c501_45b8_b3f2_494660c9aefa.slice/crio-e443731fef87154274755b4ffb0ff39aa3fe2a55b9479ef308bab95108b194bb WatchSource:0}: Error finding container e443731fef87154274755b4ffb0ff39aa3fe2a55b9479ef308bab95108b194bb: Status 404 returned error can't find the container with id e443731fef87154274755b4ffb0ff39aa3fe2a55b9479ef308bab95108b194bb Nov 03 13:58:17.058213 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:17.058148 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr"] Nov 03 13:58:17.905561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:17.905524 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" event={"ID":"47db33e4-c501-45b8-b3f2-494660c9aefa","Type":"ContainerStarted","Data":"4c488fa9ca056f00f1e100d56fa649699f304bee9841317418de000eacebd206"} Nov 03 13:58:17.905939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:17.905592 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" event={"ID":"47db33e4-c501-45b8-b3f2-494660c9aefa","Type":"ContainerStarted","Data":"e443731fef87154274755b4ffb0ff39aa3fe2a55b9479ef308bab95108b194bb"} Nov 03 13:58:18.558665 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:58:18.558631 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:58:18.869481 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:18.869396 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:18.914225 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:18.914194 4924 generic.go:356] "Generic (PLEG): container finished" podID="47db33e4-c501-45b8-b3f2-494660c9aefa" containerID="4c488fa9ca056f00f1e100d56fa649699f304bee9841317418de000eacebd206" exitCode=0 Nov 03 13:58:18.914696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:18.914241 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" event={"ID":"47db33e4-c501-45b8-b3f2-494660c9aefa","Type":"ContainerDied","Data":"4c488fa9ca056f00f1e100d56fa649699f304bee9841317418de000eacebd206"} Nov 03 13:58:19.094646 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.094608 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:19.212857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.212824 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-xks-ca\") pod \"47db33e4-c501-45b8-b3f2-494660c9aefa\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " Nov 03 13:58:19.213061 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.212875 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-openshift-ca\") pod \"47db33e4-c501-45b8-b3f2-494660c9aefa\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " Nov 03 13:58:19.213061 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.213009 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47db33e4-c501-45b8-b3f2-494660c9aefa-kube-api-access-bvd85\" (UniqueName: \"kubernetes.io/projected/47db33e4-c501-45b8-b3f2-494660c9aefa-kube-api-access-bvd85\") pod \"47db33e4-c501-45b8-b3f2-494660c9aefa\" (UID: \"47db33e4-c501-45b8-b3f2-494660c9aefa\") " Nov 03 13:58:19.213330 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.213302 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-openshift-ca" pod "47db33e4-c501-45b8-b3f2-494660c9aefa" (UID: "47db33e4-c501-45b8-b3f2-494660c9aefa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:19.213550 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.213530 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:19.218273 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.215515 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-xks-ca" pod "47db33e4-c501-45b8-b3f2-494660c9aefa" (UID: "47db33e4-c501-45b8-b3f2-494660c9aefa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:19.218273 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.215539 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47db33e4-c501-45b8-b3f2-494660c9aefa-kube-api-access-bvd85" pod "47db33e4-c501-45b8-b3f2-494660c9aefa" (UID: "47db33e4-c501-45b8-b3f2-494660c9aefa"). InnerVolumeSpecName "kube-api-access-bvd85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:19.314607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.314572 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47db33e4-c501-45b8-b3f2-494660c9aefa-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:19.314607 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.314602 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvd85\" (UniqueName: \"kubernetes.io/projected/47db33e4-c501-45b8-b3f2-494660c9aefa-kube-api-access-bvd85\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:19.921860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.921826 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" Nov 03 13:58:19.921860 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.921846 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr" event={"ID":"47db33e4-c501-45b8-b3f2-494660c9aefa","Type":"ContainerDied","Data":"e443731fef87154274755b4ffb0ff39aa3fe2a55b9479ef308bab95108b194bb"} Nov 03 13:58:19.922331 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:19.921895 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e443731fef87154274755b4ffb0ff39aa3fe2a55b9479ef308bab95108b194bb" Nov 03 13:58:23.979580 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:23.979546 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:27.865663 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:27.865627 4924 scope.go:122] "RemoveContainer" containerID="6c044aa23c81f31f38c293925ddfb591bf7c1133abd8851cc31c126970bd6ce2" Nov 03 13:58:28.778828 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:28.778795 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:29.038787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:29.038709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:34.131676 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:34.131634 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:34.816114 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:34.816057 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:39.195865 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:39.195827 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:44.393353 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:44.393309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:45.791471 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.791433 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp"] Nov 03 13:58:45.792015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.791985 4924 cpu_manager.go:414] "containerMap: removing container" podUID="47db33e4-c501-45b8-b3f2-494660c9aefa" containerName="revoke-keys" Nov 03 13:58:45.792015 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.792002 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47db33e4-c501-45b8-b3f2-494660c9aefa" containerName="revoke-keys" Nov 03 13:58:45.792181 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.792170 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="47db33e4-c501-45b8-b3f2-494660c9aefa" containerName="revoke-keys" Nov 03 13:58:45.796813 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.796789 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:45.851345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.851303 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp"] Nov 03 13:58:45.892996 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.892964 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:58:45.893168 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.893036 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:58:45.917981 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.917944 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:45.918188 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.918095 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:45.918258 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:45.918218 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnbl\" (UniqueName: \"kubernetes.io/projected/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-kube-api-access-lqnbl\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:46.018910 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:46.018875 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:46.019119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:46.018954 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnbl\" (UniqueName: \"kubernetes.io/projected/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-kube-api-access-lqnbl\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:46.019119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:46.019032 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:46.019685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:46.019660 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:46.021405 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:46.021379 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:46.088886 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:46.088810 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnbl\" (UniqueName: \"kubernetes.io/projected/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-kube-api-access-lqnbl\") pod \"maas-api-revoke-sub-7dd726f0-vcwwp\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:46.111054 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:46.111019 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:46.493689 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:58:46.493649 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b106ac_797e_43f6_bb2e_7e8a83fd8e19.slice/crio-10e2a3126b768542f05cb812e7f49c6714fdac4a5ae0e77ef1a147d8e1eca141 WatchSource:0}: Error finding container 10e2a3126b768542f05cb812e7f49c6714fdac4a5ae0e77ef1a147d8e1eca141: Status 404 returned error can't find the container with id 10e2a3126b768542f05cb812e7f49c6714fdac4a5ae0e77ef1a147d8e1eca141 Nov 03 13:58:46.500042 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:46.500000 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp"] Nov 03 13:58:47.112821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:47.112739 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" event={"ID":"64b106ac-797e-43f6-bb2e-7e8a83fd8e19","Type":"ContainerStarted","Data":"4dc2fee044d5eab85e27dde416f944ea11e0aa53c0fa26341377f7ecb875a251"} Nov 03 13:58:47.112821 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:47.112801 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" event={"ID":"64b106ac-797e-43f6-bb2e-7e8a83fd8e19","Type":"ContainerStarted","Data":"10e2a3126b768542f05cb812e7f49c6714fdac4a5ae0e77ef1a147d8e1eca141"} Nov 03 13:58:47.242891 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:47.242833 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" podStartSLOduration=2.242819474 podStartE2EDuration="2.242819474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:58:47.23866844 +0000 UTC m=+3260.433072026" watchObservedRunningTime="2026-11-03 13:58:47.242819474 +0000 UTC m=+3260.437223068" Nov 03 13:58:48.121082 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:48.121032 4924 generic.go:356] "Generic (PLEG): container finished" podID="64b106ac-797e-43f6-bb2e-7e8a83fd8e19" containerID="4dc2fee044d5eab85e27dde416f944ea11e0aa53c0fa26341377f7ecb875a251" exitCode=0 Nov 03 13:58:48.121538 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:48.121114 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" event={"ID":"64b106ac-797e-43f6-bb2e-7e8a83fd8e19","Type":"ContainerDied","Data":"4dc2fee044d5eab85e27dde416f944ea11e0aa53c0fa26341377f7ecb875a251"} Nov 03 13:58:48.561300 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:58:48.561257 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:58:48.975346 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:48.975306 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:49.350186 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.350159 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:49.456746 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.456703 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-kube-api-access-lqnbl\" (UniqueName: \"kubernetes.io/projected/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-kube-api-access-lqnbl\") pod \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " Nov 03 13:58:49.456968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.456853 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-openshift-ca\") pod \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " Nov 03 13:58:49.456968 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.456906 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-xks-ca\") pod \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\" (UID: \"64b106ac-797e-43f6-bb2e-7e8a83fd8e19\") " Nov 03 13:58:49.457264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.457238 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-openshift-ca" pod "64b106ac-797e-43f6-bb2e-7e8a83fd8e19" (UID: "64b106ac-797e-43f6-bb2e-7e8a83fd8e19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:49.457526 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.457501 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:49.459561 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.459533 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-xks-ca" pod "64b106ac-797e-43f6-bb2e-7e8a83fd8e19" (UID: "64b106ac-797e-43f6-bb2e-7e8a83fd8e19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:49.460139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.460114 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-kube-api-access-lqnbl" pod "64b106ac-797e-43f6-bb2e-7e8a83fd8e19" (UID: "64b106ac-797e-43f6-bb2e-7e8a83fd8e19"). InnerVolumeSpecName "kube-api-access-lqnbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:49.558978 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.558884 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnbl\" (UniqueName: \"kubernetes.io/projected/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-kube-api-access-lqnbl\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:49.558978 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:49.558926 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b106ac-797e-43f6-bb2e-7e8a83fd8e19-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:50.137200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:50.137167 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" Nov 03 13:58:50.137364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:50.137205 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp" event={"ID":"64b106ac-797e-43f6-bb2e-7e8a83fd8e19","Type":"ContainerDied","Data":"10e2a3126b768542f05cb812e7f49c6714fdac4a5ae0e77ef1a147d8e1eca141"} Nov 03 13:58:50.137364 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:50.137264 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10e2a3126b768542f05cb812e7f49c6714fdac4a5ae0e77ef1a147d8e1eca141" Nov 03 13:58:50.770777 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:50.770717 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:52.645108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:52.645060 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:53.277124 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:53.274932 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h"] Nov 03 13:58:53.297630 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:53.297599 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:53.457095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:53.455500 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:53.928912 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:53.928874 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:54.592421 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:54.592125 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:54.719101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:54.718043 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:56.151993 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.151956 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:56.292644 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.292608 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:56.498123 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.498090 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:56.648553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.648516 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9"] Nov 03 13:58:56.649021 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.649008 4924 cpu_manager.go:414] "containerMap: removing container" podUID="64b106ac-797e-43f6-bb2e-7e8a83fd8e19" containerName="revoke-keys" Nov 03 13:58:56.649087 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.649022 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b106ac-797e-43f6-bb2e-7e8a83fd8e19" containerName="revoke-keys" Nov 03 13:58:56.649146 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.649136 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b106ac-797e-43f6-bb2e-7e8a83fd8e19" containerName="revoke-keys" Nov 03 13:58:56.653395 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.653378 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.710935 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.710908 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:58:56.711161 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.711147 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:58:56.751500 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.751414 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:56.759372 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.759341 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.759501 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.759388 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.759590 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.759574 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztnqj\" (UniqueName: \"kubernetes.io/projected/6aa9965c-54b4-4492-9dcf-ba27e74c028e-kube-api-access-ztnqj\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.761362 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.761341 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9"] Nov 03 13:58:56.860857 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.860822 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztnqj\" (UniqueName: \"kubernetes.io/projected/6aa9965c-54b4-4492-9dcf-ba27e74c028e-kube-api-access-ztnqj\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.861030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.860962 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.861030 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.860987 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.861731 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.861702 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.863397 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.863374 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.918984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.918949 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztnqj\" (UniqueName: \"kubernetes.io/projected/6aa9965c-54b4-4492-9dcf-ba27e74c028e-kube-api-access-ztnqj\") pod \"maas-api-revoke-sub-d4274dd0-msdc9\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:56.967116 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:56.967064 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:58:57.343128 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:57.343094 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9"] Nov 03 13:58:57.343540 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:58:57.343515 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa9965c_54b4_4492_9dcf_ba27e74c028e.slice/crio-68c620274a78f1c0564393fb66244470fe95d8f59035c08d88c9ae815656b8c7 WatchSource:0}: Error finding container 68c620274a78f1c0564393fb66244470fe95d8f59035c08d88c9ae815656b8c7: Status 404 returned error can't find the container with id 68c620274a78f1c0564393fb66244470fe95d8f59035c08d88c9ae815656b8c7 Nov 03 13:58:58.198057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:58.198024 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" event={"ID":"6aa9965c-54b4-4492-9dcf-ba27e74c028e","Type":"ContainerStarted","Data":"61b2c530ce89c6684902771b17d2fd211e00c341b55257085e32107d0221f49e"} Nov 03 13:58:58.198243 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:58.198095 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" event={"ID":"6aa9965c-54b4-4492-9dcf-ba27e74c028e","Type":"ContainerStarted","Data":"68c620274a78f1c0564393fb66244470fe95d8f59035c08d88c9ae815656b8c7"} Nov 03 13:58:58.309394 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:58.309336 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" podStartSLOduration=2.309323718 podStartE2EDuration="2.309323718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:58:58.305992662 +0000 UTC m=+3271.500396274" watchObservedRunningTime="2026-11-03 13:58:58.309323718 +0000 UTC m=+3271.503727312" Nov 03 13:58:58.429719 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:58.429683 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:58.557604 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:58.557519 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:58:59.206717 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:59.206671 4924 generic.go:356] "Generic (PLEG): container finished" podID="6aa9965c-54b4-4492-9dcf-ba27e74c028e" containerID="61b2c530ce89c6684902771b17d2fd211e00c341b55257085e32107d0221f49e" exitCode=0 Nov 03 13:58:59.206901 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:58:59.206758 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" event={"ID":"6aa9965c-54b4-4492-9dcf-ba27e74c028e","Type":"ContainerDied","Data":"61b2c530ce89c6684902771b17d2fd211e00c341b55257085e32107d0221f49e"} Nov 03 13:59:00.366982 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.366953 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:59:00.501393 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.501302 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-xks-ca\") pod \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " Nov 03 13:59:00.501393 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.501357 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-openshift-ca\") pod \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " Nov 03 13:59:00.501645 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.501446 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa9965c-54b4-4492-9dcf-ba27e74c028e-kube-api-access-ztnqj\" (UniqueName: \"kubernetes.io/projected/6aa9965c-54b4-4492-9dcf-ba27e74c028e-kube-api-access-ztnqj\") pod \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\" (UID: \"6aa9965c-54b4-4492-9dcf-ba27e74c028e\") " Nov 03 13:59:00.501853 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.501808 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-openshift-ca" pod "6aa9965c-54b4-4492-9dcf-ba27e74c028e" (UID: "6aa9965c-54b4-4492-9dcf-ba27e74c028e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:59:00.503714 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.503686 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-xks-ca" pod "6aa9965c-54b4-4492-9dcf-ba27e74c028e" (UID: "6aa9965c-54b4-4492-9dcf-ba27e74c028e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:59:00.504066 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.504029 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa9965c-54b4-4492-9dcf-ba27e74c028e-kube-api-access-ztnqj" pod "6aa9965c-54b4-4492-9dcf-ba27e74c028e" (UID: "6aa9965c-54b4-4492-9dcf-ba27e74c028e"). InnerVolumeSpecName "kube-api-access-ztnqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:00.603469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.603425 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:00.603469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.603463 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa9965c-54b4-4492-9dcf-ba27e74c028e-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:00.603469 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:00.603474 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztnqj\" (UniqueName: \"kubernetes.io/projected/6aa9965c-54b4-4492-9dcf-ba27e74c028e-kube-api-access-ztnqj\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:01.221906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:01.221817 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" Nov 03 13:59:01.221906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:01.221841 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9" event={"ID":"6aa9965c-54b4-4492-9dcf-ba27e74c028e","Type":"ContainerDied","Data":"68c620274a78f1c0564393fb66244470fe95d8f59035c08d88c9ae815656b8c7"} Nov 03 13:59:01.221906 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:01.221888 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c620274a78f1c0564393fb66244470fe95d8f59035c08d88c9ae815656b8c7" Nov 03 13:59:01.568290 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:01.568206 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:06.673579 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:06.673544 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:08.320392 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.320349 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" podUID="6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" containerName="main" containerID="cri-o://9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b" gracePeriod=45 Nov 03 13:59:08.596108 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.596083 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:59:08.694610 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.694580 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-model-cache\") pod \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " Nov 03 13:59:08.694777 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.694641 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tmp-dir\") pod \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " Nov 03 13:59:08.694819 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.694776 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-dshm\") pod \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " Nov 03 13:59:08.694867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.694851 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-home\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-home\") pod \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " Nov 03 13:59:08.694961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.694904 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-kube-api-access-cr2nt\" (UniqueName: \"kubernetes.io/projected/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-kube-api-access-cr2nt\") pod \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " Nov 03 13:59:08.694961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.694850 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-model-cache" pod "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" (UID: "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:08.694961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.694933 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tls-certs\") pod \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\" (UID: \"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f\") " Nov 03 13:59:08.694961 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.694940 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tmp-dir" pod "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" (UID: "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:08.695204 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.695030 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-home" pod "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" (UID: "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:08.695350 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.695330 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:08.695350 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.695350 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:08.695466 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.695359 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:08.696966 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.696943 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-dshm" pod "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" (UID: "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:08.697065 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.697042 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-kube-api-access-cr2nt" pod "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" (UID: "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f"). InnerVolumeSpecName "kube-api-access-cr2nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:08.697130 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.697061 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tls-certs" pod "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" (UID: "6a7925bb-39c1-4877-8428-5fe2f7ec5a9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:59:08.796052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.796017 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:08.796052 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.796053 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr2nt\" (UniqueName: \"kubernetes.io/projected/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-kube-api-access-cr2nt\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:08.796298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:08.796082 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:09.281593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.281498 4924 generic.go:356] "Generic (PLEG): container finished" podID="6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" containerID="9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b" exitCode=0 Nov 03 13:59:09.281593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.281551 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" Nov 03 13:59:09.281859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.281556 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" event={"ID":"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f","Type":"ContainerDied","Data":"9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b"} Nov 03 13:59:09.281859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.281663 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" event={"ID":"6a7925bb-39c1-4877-8428-5fe2f7ec5a9f","Type":"ContainerDied","Data":"bbbc2fea35d3009c8b16b91aabe0b7a4a284221d7954b7f28de7470faf499186"} Nov 03 13:59:09.281859 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.281680 4924 scope.go:122] "RemoveContainer" containerID="9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b" Nov 03 13:59:09.294326 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.294304 4924 scope.go:122] "RemoveContainer" containerID="9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b" Nov 03 13:59:09.294653 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:09.294626 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b\": container with ID starting with 9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b not found: ID does not exist" containerID="9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b" Nov 03 13:59:09.294740 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.294665 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b"} err="failed to get container status \"9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b\": rpc error: code = NotFound desc = could not find container \"9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b\": container with ID starting with 9f10b987fa497429c1035cf7deb8056900f77f18eabb53403f3b56066dfc301b not found: ID does not exist" Nov 03 13:59:09.436524 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.436486 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h"] Nov 03 13:59:09.483384 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.483340 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h"] Nov 03 13:59:09.568295 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:09.568204 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-2bceb1/auth-test-model-2bceb1-kserve-75f56dbccc-4f94h" podUID="6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.123:8000/ready\": context deadline exceeded" Nov 03 13:59:10.951264 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:10.951229 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" path="/var/lib/kubelet/pods/6a7925bb-39c1-4877-8428-5fe2f7ec5a9f/volumes" Nov 03 13:59:11.792037 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:11.791997 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:16.851100 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:16.851035 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:18.558201 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:18.558167 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:59:21.884891 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:21.884855 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:24.123033 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.122994 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-674d22"] Nov 03 13:59:24.123509 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.123493 4924 cpu_manager.go:414] "containerMap: removing container" podUID="6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" containerName="main" Nov 03 13:59:24.123553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.123510 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" containerName="main" Nov 03 13:59:24.123553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.123532 4924 cpu_manager.go:414] "containerMap: removing container" podUID="6aa9965c-54b4-4492-9dcf-ba27e74c028e" containerName="revoke-keys" Nov 03 13:59:24.123553 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.123537 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa9965c-54b4-4492-9dcf-ba27e74c028e" containerName="revoke-keys" Nov 03 13:59:24.123657 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.123616 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7925bb-39c1-4877-8428-5fe2f7ec5a9f" containerName="main" Nov 03 13:59:24.123657 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.123627 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa9965c-54b4-4492-9dcf-ba27e74c028e" containerName="revoke-keys" Nov 03 13:59:24.128345 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.128324 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:24.179826 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.179785 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-674d22"] Nov 03 13:59:24.183357 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.183326 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t22jv\" (UniqueName: \"kubernetes.io/projected/4142fd84-ddef-4858-be47-7506e5836949-kube-api-access-t22jv\") pod \"test-curl-7600-674d22\" (UID: \"4142fd84-ddef-4858-be47-7506e5836949\") " pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:24.221926 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.221895 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kmbtk\"" Nov 03 13:59:24.284487 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.284445 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22jv\" (UniqueName: \"kubernetes.io/projected/4142fd84-ddef-4858-be47-7506e5836949-kube-api-access-t22jv\") pod \"test-curl-7600-674d22\" (UID: \"4142fd84-ddef-4858-be47-7506e5836949\") " pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:24.338391 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.338359 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t22jv\" (UniqueName: \"kubernetes.io/projected/4142fd84-ddef-4858-be47-7506e5836949-kube-api-access-t22jv\") pod \"test-curl-7600-674d22\" (UID: \"4142fd84-ddef-4858-be47-7506e5836949\") " pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:24.440271 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.440234 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:24.810558 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:59:24.810529 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4142fd84_ddef_4858_be47_7506e5836949.slice/crio-7dc9b4505c1905dfadf8ee9747ab214b26b7e281168f2886000ffe6d74ffceca WatchSource:0}: Error finding container 7dc9b4505c1905dfadf8ee9747ab214b26b7e281168f2886000ffe6d74ffceca: Status 404 returned error can't find the container with id 7dc9b4505c1905dfadf8ee9747ab214b26b7e281168f2886000ffe6d74ffceca Nov 03 13:59:24.811222 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:24.811198 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-674d22"] Nov 03 13:59:25.409738 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:25.406793 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-674d22" event={"ID":"4142fd84-ddef-4858-be47-7506e5836949","Type":"ContainerStarted","Data":"4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43"} Nov 03 13:59:25.409738 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:25.406854 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-674d22" event={"ID":"4142fd84-ddef-4858-be47-7506e5836949","Type":"ContainerStarted","Data":"7dc9b4505c1905dfadf8ee9747ab214b26b7e281168f2886000ffe6d74ffceca"} Nov 03 13:59:25.424811 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:25.424770 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:25.538101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:25.532885 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-674d22" podStartSLOduration=1.532860807 podStartE2EDuration="1.532860807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:25.512581155 +0000 UTC m=+3298.706984746" watchObservedRunningTime="2026-11-03 13:59:25.532860807 +0000 UTC m=+3298.727264399" Nov 03 13:59:26.910939 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:26.910906 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:26.957099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:26.957041 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-674d22"] Nov 03 13:59:26.960876 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:26.960849 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-674d22"] Nov 03 13:59:27.376281 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.376205 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-da508d"] Nov 03 13:59:27.376725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.376709 4924 cpu_manager.go:414] "containerMap: removing container" podUID="4142fd84-ddef-4858-be47-7506e5836949" containerName="test-curl-7600-674d22" Nov 03 13:59:27.376725 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.376725 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4142fd84-ddef-4858-be47-7506e5836949" containerName="test-curl-7600-674d22" Nov 03 13:59:27.376870 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.376860 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4142fd84-ddef-4858-be47-7506e5836949" containerName="test-curl-7600-674d22" Nov 03 13:59:27.379864 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.379844 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:27.420346 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.420309 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprrv\" (UniqueName: \"kubernetes.io/projected/91b90276-c2cb-4ff2-9430-272cb113612f-kube-api-access-mprrv\") pod \"test-curl-7600-da508d\" (UID: \"91b90276-c2cb-4ff2-9430-272cb113612f\") " pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:27.420773 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.420744 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-674d22" podUID="4142fd84-ddef-4858-be47-7506e5836949" containerName="test-curl-7600-674d22" containerID="cri-o://4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43" gracePeriod=2 Nov 03 13:59:27.429594 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.429570 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-da508d"] Nov 03 13:59:27.486629 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.486592 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:59:27.496881 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.496273 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 13:59:27.521316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.521280 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprrv\" (UniqueName: \"kubernetes.io/projected/91b90276-c2cb-4ff2-9430-272cb113612f-kube-api-access-mprrv\") pod \"test-curl-7600-da508d\" (UID: \"91b90276-c2cb-4ff2-9430-272cb113612f\") " pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:27.530360 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:27.530325 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-674d22\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="4142fd84-ddef-4858-be47-7506e5836949" pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:27.575627 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.575595 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprrv\" (UniqueName: \"kubernetes.io/projected/91b90276-c2cb-4ff2-9430-272cb113612f-kube-api-access-mprrv\") pod \"test-curl-7600-da508d\" (UID: \"91b90276-c2cb-4ff2-9430-272cb113612f\") " pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:27.694439 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:27.694399 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:28.071213 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:28.071182 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-da508d"] Nov 03 13:59:28.072050 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:59:28.072026 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91b90276_c2cb_4ff2_9430_272cb113612f.slice/crio-6cf70cad55d0432c2aeea0cf9d143564d79f75de57c557143d146f5e66a0b03c WatchSource:0}: Error finding container 6cf70cad55d0432c2aeea0cf9d143564d79f75de57c557143d146f5e66a0b03c: Status 404 returned error can't find the container with id 6cf70cad55d0432c2aeea0cf9d143564d79f75de57c557143d146f5e66a0b03c Nov 03 13:59:28.429316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:28.429286 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-da508d" event={"ID":"91b90276-c2cb-4ff2-9430-272cb113612f","Type":"ContainerStarted","Data":"910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae"} Nov 03 13:59:28.429528 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:28.429324 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-da508d" event={"ID":"91b90276-c2cb-4ff2-9430-272cb113612f","Type":"ContainerStarted","Data":"6cf70cad55d0432c2aeea0cf9d143564d79f75de57c557143d146f5e66a0b03c"} Nov 03 13:59:28.536411 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:28.536358 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-da508d" podStartSLOduration=1.536343852 podStartE2EDuration="1.536343852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:28.533953045 +0000 UTC m=+3301.728356638" watchObservedRunningTime="2026-11-03 13:59:28.536343852 +0000 UTC m=+3301.730747445" Nov 03 13:59:28.581496 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:28.581459 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-674d22\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="4142fd84-ddef-4858-be47-7506e5836949" pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:29.628898 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:29.628870 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:29.748470 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:29.748376 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4142fd84-ddef-4858-be47-7506e5836949-kube-api-access-t22jv\" (UniqueName: \"kubernetes.io/projected/4142fd84-ddef-4858-be47-7506e5836949-kube-api-access-t22jv\") pod \"4142fd84-ddef-4858-be47-7506e5836949\" (UID: \"4142fd84-ddef-4858-be47-7506e5836949\") " Nov 03 13:59:29.750483 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:29.750449 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4142fd84-ddef-4858-be47-7506e5836949-kube-api-access-t22jv" pod "4142fd84-ddef-4858-be47-7506e5836949" (UID: "4142fd84-ddef-4858-be47-7506e5836949"). InnerVolumeSpecName "kube-api-access-t22jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:29.850245 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:29.850203 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t22jv\" (UniqueName: \"kubernetes.io/projected/4142fd84-ddef-4858-be47-7506e5836949-kube-api-access-t22jv\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:29.877593 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:29.877555 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-da508d"] Nov 03 13:59:29.884431 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:29.884400 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-da508d"] Nov 03 13:59:30.045176 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.045097 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-9929d1"] Nov 03 13:59:30.045588 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.045576 4924 cpu_manager.go:414] "containerMap: removing container" podUID="91b90276-c2cb-4ff2-9430-272cb113612f" containerName="test-curl-7600-da508d" Nov 03 13:59:30.045631 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.045590 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b90276-c2cb-4ff2-9430-272cb113612f" containerName="test-curl-7600-da508d" Nov 03 13:59:30.045702 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.045693 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b90276-c2cb-4ff2-9430-272cb113612f" containerName="test-curl-7600-da508d" Nov 03 13:59:30.049007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.048990 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:30.101592 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.101556 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-9929d1"] Nov 03 13:59:30.153251 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.153215 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btffx\" (UniqueName: \"kubernetes.io/projected/c3449968-1aea-4b5d-b7da-6cd5fd0e9fee-kube-api-access-btffx\") pod \"test-curl-7600-9929d1\" (UID: \"c3449968-1aea-4b5d-b7da-6cd5fd0e9fee\") " pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:30.254635 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.254599 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btffx\" (UniqueName: \"kubernetes.io/projected/c3449968-1aea-4b5d-b7da-6cd5fd0e9fee-kube-api-access-btffx\") pod \"test-curl-7600-9929d1\" (UID: \"c3449968-1aea-4b5d-b7da-6cd5fd0e9fee\") " pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:30.308567 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.308487 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btffx\" (UniqueName: \"kubernetes.io/projected/c3449968-1aea-4b5d-b7da-6cd5fd0e9fee-kube-api-access-btffx\") pod \"test-curl-7600-9929d1\" (UID: \"c3449968-1aea-4b5d-b7da-6cd5fd0e9fee\") " pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:30.362007 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.361977 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:30.451099 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.451050 4924 generic.go:356] "Generic (PLEG): container finished" podID="4142fd84-ddef-4858-be47-7506e5836949" containerID="4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43" exitCode=137 Nov 03 13:59:30.451284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.451114 4924 scope.go:122] "RemoveContainer" containerID="4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43" Nov 03 13:59:30.451284 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.451126 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:30.451446 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.451422 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-da508d" podUID="91b90276-c2cb-4ff2-9430-272cb113612f" containerName="test-curl-7600-da508d" containerID="cri-o://910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae" gracePeriod=2 Nov 03 13:59:30.464432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.463138 4924 scope.go:122] "RemoveContainer" containerID="4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43" Nov 03 13:59:30.464432 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:30.463464 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43\": container with ID starting with 4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43 not found: ID does not exist" containerID="4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43" Nov 03 13:59:30.464432 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.463499 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43"} err="failed to get container status \"4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43\": rpc error: code = NotFound desc = could not find container \"4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43\": container with ID starting with 4cf8b93739a5b7ea46cf7ea8325ba161b6c4c3a4f7951628580b99be44741f43 not found: ID does not exist" Nov 03 13:59:30.499209 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:30.499171 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-da508d\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="91b90276-c2cb-4ff2-9430-272cb113612f" pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:30.547062 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:30.547010 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-da508d\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="91b90276-c2cb-4ff2-9430-272cb113612f" pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:30.595033 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:30.594948 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-674d22\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="4142fd84-ddef-4858-be47-7506e5836949" pod="opendatahub/test-curl-7600-674d22" Nov 03 13:59:30.732835 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:59:30.732801 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3449968_1aea_4b5d_b7da_6cd5fd0e9fee.slice/crio-58e0ef651bda8f5cf94930117aea654cff3d1f2537ff147a2c2516de83a1ed3c WatchSource:0}: Error finding container 58e0ef651bda8f5cf94930117aea654cff3d1f2537ff147a2c2516de83a1ed3c: Status 404 returned error can't find the container with id 58e0ef651bda8f5cf94930117aea654cff3d1f2537ff147a2c2516de83a1ed3c Nov 03 13:59:30.733263 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.733237 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-9929d1"] Nov 03 13:59:30.951834 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:30.951797 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4142fd84-ddef-4858-be47-7506e5836949" path="/var/lib/kubelet/pods/4142fd84-ddef-4858-be47-7506e5836949/volumes" Nov 03 13:59:31.461353 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:31.461310 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-9929d1" event={"ID":"c3449968-1aea-4b5d-b7da-6cd5fd0e9fee","Type":"ContainerStarted","Data":"bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea"} Nov 03 13:59:31.461568 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:31.461363 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-9929d1" event={"ID":"c3449968-1aea-4b5d-b7da-6cd5fd0e9fee","Type":"ContainerStarted","Data":"58e0ef651bda8f5cf94930117aea654cff3d1f2537ff147a2c2516de83a1ed3c"} Nov 03 13:59:31.579413 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:31.579357 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-9929d1" podStartSLOduration=1.579341409 podStartE2EDuration="1.579341409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:31.578843022 +0000 UTC m=+3304.773246616" watchObservedRunningTime="2026-11-03 13:59:31.579341409 +0000 UTC m=+3304.773745003" Nov 03 13:59:31.962063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:31.962031 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:32.710200 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:32.710141 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:32.784327 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:32.784296 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b90276-c2cb-4ff2-9430-272cb113612f-kube-api-access-mprrv\" (UniqueName: \"kubernetes.io/projected/91b90276-c2cb-4ff2-9430-272cb113612f-kube-api-access-mprrv\") pod \"91b90276-c2cb-4ff2-9430-272cb113612f\" (UID: \"91b90276-c2cb-4ff2-9430-272cb113612f\") " Nov 03 13:59:32.786373 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:32.786342 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b90276-c2cb-4ff2-9430-272cb113612f-kube-api-access-mprrv" pod "91b90276-c2cb-4ff2-9430-272cb113612f" (UID: "91b90276-c2cb-4ff2-9430-272cb113612f"). InnerVolumeSpecName "kube-api-access-mprrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:32.886036 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:32.885997 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mprrv\" (UniqueName: \"kubernetes.io/projected/91b90276-c2cb-4ff2-9430-272cb113612f-kube-api-access-mprrv\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:32.953212 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:32.953175 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b90276-c2cb-4ff2-9430-272cb113612f" path="/var/lib/kubelet/pods/91b90276-c2cb-4ff2-9430-272cb113612f/volumes" Nov 03 13:59:33.034059 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.034024 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-9929d1"] Nov 03 13:59:33.041647 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.041619 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-9929d1"] Nov 03 13:59:33.207293 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.207260 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-e187fd"] Nov 03 13:59:33.207787 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.207774 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" containerName="test-curl-7600-9929d1" Nov 03 13:59:33.207827 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.207789 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" containerName="test-curl-7600-9929d1" Nov 03 13:59:33.207884 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.207874 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" containerName="test-curl-7600-9929d1" Nov 03 13:59:33.211252 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.211235 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:33.264783 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.264741 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-e187fd"] Nov 03 13:59:33.289645 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.289599 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6wqb\" (UniqueName: \"kubernetes.io/projected/96dfab60-4888-41a2-a823-2cbd19d153ea-kube-api-access-x6wqb\") pod \"test-curl-7600-e187fd\" (UID: \"96dfab60-4888-41a2-a823-2cbd19d153ea\") " pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:33.390666 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.390630 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6wqb\" (UniqueName: \"kubernetes.io/projected/96dfab60-4888-41a2-a823-2cbd19d153ea-kube-api-access-x6wqb\") pod \"test-curl-7600-e187fd\" (UID: \"96dfab60-4888-41a2-a823-2cbd19d153ea\") " pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:33.445029 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.444996 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6wqb\" (UniqueName: \"kubernetes.io/projected/96dfab60-4888-41a2-a823-2cbd19d153ea-kube-api-access-x6wqb\") pod \"test-curl-7600-e187fd\" (UID: \"96dfab60-4888-41a2-a823-2cbd19d153ea\") " pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:33.479019 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.478945 4924 generic.go:356] "Generic (PLEG): container finished" podID="91b90276-c2cb-4ff2-9430-272cb113612f" containerID="910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae" exitCode=137 Nov 03 13:59:33.479019 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.478997 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-da508d" Nov 03 13:59:33.479253 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.479031 4924 scope.go:122] "RemoveContainer" containerID="910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae" Nov 03 13:59:33.479253 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.479215 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-9929d1" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" containerName="test-curl-7600-9929d1" containerID="cri-o://bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea" gracePeriod=2 Nov 03 13:59:33.488390 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.488360 4924 scope.go:122] "RemoveContainer" containerID="910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae" Nov 03 13:59:33.488670 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:33.488649 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae\": container with ID starting with 910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae not found: ID does not exist" containerID="910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae" Nov 03 13:59:33.488744 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.488682 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae"} err="failed to get container status \"910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae\": rpc error: code = NotFound desc = could not find container \"910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae\": container with ID starting with 910631e2b19d694a70720849154b8219f8939866eb011e1836d338bb09cb1aae not found: ID does not exist" Nov 03 13:59:33.523666 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.523637 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:33.528285 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:33.528252 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-9929d1\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:33.576011 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:33.575966 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-9929d1\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:33.896822 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:33.896793 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-e187fd"] Nov 03 13:59:33.898127 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:59:33.898097 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96dfab60_4888_41a2_a823_2cbd19d153ea.slice/crio-7657c093dcbe248f18aefd8eeba4e59aba5a84e80925655817a3cb62fd05d560 WatchSource:0}: Error finding container 7657c093dcbe248f18aefd8eeba4e59aba5a84e80925655817a3cb62fd05d560: Status 404 returned error can't find the container with id 7657c093dcbe248f18aefd8eeba4e59aba5a84e80925655817a3cb62fd05d560 Nov 03 13:59:34.487277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:34.487239 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-e187fd" event={"ID":"96dfab60-4888-41a2-a823-2cbd19d153ea","Type":"ContainerStarted","Data":"997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf"} Nov 03 13:59:34.487277 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:34.487288 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-e187fd" event={"ID":"96dfab60-4888-41a2-a823-2cbd19d153ea","Type":"ContainerStarted","Data":"7657c093dcbe248f18aefd8eeba4e59aba5a84e80925655817a3cb62fd05d560"} Nov 03 13:59:34.535022 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:34.534983 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-9929d1\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:34.644552 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:34.644488 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-e187fd" podStartSLOduration=1.6444703760000001 podStartE2EDuration="1.644470376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:34.64003955 +0000 UTC m=+3307.834443145" watchObservedRunningTime="2026-11-03 13:59:34.644470376 +0000 UTC m=+3307.838873972" Nov 03 13:59:35.134534 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:35.134497 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:35.304525 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:35.304487 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:35.724790 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:35.724764 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:35.820106 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:35.819999 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3449968-1aea-4b5d-b7da-6cd5fd0e9fee-kube-api-access-btffx\" (UniqueName: \"kubernetes.io/projected/c3449968-1aea-4b5d-b7da-6cd5fd0e9fee-kube-api-access-btffx\") pod \"c3449968-1aea-4b5d-b7da-6cd5fd0e9fee\" (UID: \"c3449968-1aea-4b5d-b7da-6cd5fd0e9fee\") " Nov 03 13:59:35.822034 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:35.822012 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3449968-1aea-4b5d-b7da-6cd5fd0e9fee-kube-api-access-btffx" pod "c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" (UID: "c3449968-1aea-4b5d-b7da-6cd5fd0e9fee"). InnerVolumeSpecName "kube-api-access-btffx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:35.921220 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:35.921187 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btffx\" (UniqueName: \"kubernetes.io/projected/c3449968-1aea-4b5d-b7da-6cd5fd0e9fee-kube-api-access-btffx\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:35.998659 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:35.998623 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-e187fd"] Nov 03 13:59:36.005899 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.005866 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-e187fd"] Nov 03 13:59:36.158532 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.158483 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-082668"] Nov 03 13:59:36.159272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.159251 4924 cpu_manager.go:414] "containerMap: removing container" podUID="96dfab60-4888-41a2-a823-2cbd19d153ea" containerName="test-curl-7600-e187fd" Nov 03 13:59:36.159272 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.159272 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96dfab60-4888-41a2-a823-2cbd19d153ea" containerName="test-curl-7600-e187fd" Nov 03 13:59:36.159465 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.159451 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="96dfab60-4888-41a2-a823-2cbd19d153ea" containerName="test-curl-7600-e187fd" Nov 03 13:59:36.163201 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.163177 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:36.214242 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.214208 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-082668"] Nov 03 13:59:36.223499 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.223465 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pb9\" (UniqueName: \"kubernetes.io/projected/593060c0-d51a-43a0-993a-a3bae45ef7a3-kube-api-access-k8pb9\") pod \"test-curl-7600-082668\" (UID: \"593060c0-d51a-43a0-993a-a3bae45ef7a3\") " pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:36.324971 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.324936 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pb9\" (UniqueName: \"kubernetes.io/projected/593060c0-d51a-43a0-993a-a3bae45ef7a3-kube-api-access-k8pb9\") pod \"test-curl-7600-082668\" (UID: \"593060c0-d51a-43a0-993a-a3bae45ef7a3\") " pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:36.378943 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.378906 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pb9\" (UniqueName: \"kubernetes.io/projected/593060c0-d51a-43a0-993a-a3bae45ef7a3-kube-api-access-k8pb9\") pod \"test-curl-7600-082668\" (UID: \"593060c0-d51a-43a0-993a-a3bae45ef7a3\") " pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:36.478317 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.478131 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:36.513720 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.513680 4924 generic.go:356] "Generic (PLEG): container finished" podID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" containerID="bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea" exitCode=137 Nov 03 13:59:36.513895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.513737 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:36.513895 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.513762 4924 scope.go:122] "RemoveContainer" containerID="bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea" Nov 03 13:59:36.514005 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.513978 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-e187fd" podUID="96dfab60-4888-41a2-a823-2cbd19d153ea" containerName="test-curl-7600-e187fd" containerID="cri-o://997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf" gracePeriod=2 Nov 03 13:59:36.527063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.526517 4924 scope.go:122] "RemoveContainer" containerID="bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea" Nov 03 13:59:36.527063 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:36.526901 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea\": container with ID starting with bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea not found: ID does not exist" containerID="bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea" Nov 03 13:59:36.527063 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.526944 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea"} err="failed to get container status \"bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea\": rpc error: code = NotFound desc = could not find container \"bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea\": container with ID starting with bb415a1fb0cdc3abb4e1ff6d3902082938b91145121b0aa7455059c87a84dbea not found: ID does not exist" Nov 03 13:59:36.562118 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:36.561897 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-e187fd\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="96dfab60-4888-41a2-a823-2cbd19d153ea" pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:36.609756 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:36.609714 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-9929d1\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:36.657828 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:36.657784 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-e187fd\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="96dfab60-4888-41a2-a823-2cbd19d153ea" pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:36.858544 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:59:36.858514 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod593060c0_d51a_43a0_993a_a3bae45ef7a3.slice/crio-172b2b609c1e8aba774b89d9e77487c3f1f696745511b507380b83ccf882884e WatchSource:0}: Error finding container 172b2b609c1e8aba774b89d9e77487c3f1f696745511b507380b83ccf882884e: Status 404 returned error can't find the container with id 172b2b609c1e8aba774b89d9e77487c3f1f696745511b507380b83ccf882884e Nov 03 13:59:36.858972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.858783 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-082668"] Nov 03 13:59:36.951879 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:36.951833 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" path="/var/lib/kubelet/pods/c3449968-1aea-4b5d-b7da-6cd5fd0e9fee/volumes" Nov 03 13:59:36.995394 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:36.995348 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-e187fd\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="96dfab60-4888-41a2-a823-2cbd19d153ea" pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:37.002468 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:37.002434 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:37.043182 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:37.043142 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-9929d1\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="c3449968-1aea-4b5d-b7da-6cd5fd0e9fee" pod="opendatahub/test-curl-7600-9929d1" Nov 03 13:59:37.521970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:37.521932 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-082668" event={"ID":"593060c0-d51a-43a0-993a-a3bae45ef7a3","Type":"ContainerStarted","Data":"2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116"} Nov 03 13:59:37.521970 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:37.521971 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-082668" event={"ID":"593060c0-d51a-43a0-993a-a3bae45ef7a3","Type":"ContainerStarted","Data":"172b2b609c1e8aba774b89d9e77487c3f1f696745511b507380b83ccf882884e"} Nov 03 13:59:37.631310 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:37.631243 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-082668" podStartSLOduration=1.631227765 podStartE2EDuration="1.631227765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:37.627877353 +0000 UTC m=+3310.822280948" watchObservedRunningTime="2026-11-03 13:59:37.631227765 +0000 UTC m=+3310.825631358" Nov 03 13:59:38.853808 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:38.853776 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:38.956599 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:38.956570 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96dfab60-4888-41a2-a823-2cbd19d153ea-kube-api-access-x6wqb\" (UniqueName: \"kubernetes.io/projected/96dfab60-4888-41a2-a823-2cbd19d153ea-kube-api-access-x6wqb\") pod \"96dfab60-4888-41a2-a823-2cbd19d153ea\" (UID: \"96dfab60-4888-41a2-a823-2cbd19d153ea\") " Nov 03 13:59:38.958699 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:38.958675 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96dfab60-4888-41a2-a823-2cbd19d153ea-kube-api-access-x6wqb" pod "96dfab60-4888-41a2-a823-2cbd19d153ea" (UID: "96dfab60-4888-41a2-a823-2cbd19d153ea"). InnerVolumeSpecName "kube-api-access-x6wqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:39.057685 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.057650 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6wqb\" (UniqueName: \"kubernetes.io/projected/96dfab60-4888-41a2-a823-2cbd19d153ea-kube-api-access-x6wqb\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:39.182325 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.182283 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-082668"] Nov 03 13:59:39.186479 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.186448 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-082668"] Nov 03 13:59:39.549141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.541247 4924 generic.go:356] "Generic (PLEG): container finished" podID="96dfab60-4888-41a2-a823-2cbd19d153ea" containerID="997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf" exitCode=137 Nov 03 13:59:39.549141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.541590 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-082668" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" containerName="test-curl-7600-082668" containerID="cri-o://2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116" gracePeriod=2 Nov 03 13:59:39.549141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.541934 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-e187fd" Nov 03 13:59:39.549141 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.542375 4924 scope.go:122] "RemoveContainer" containerID="997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf" Nov 03 13:59:39.581999 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.581973 4924 scope.go:122] "RemoveContainer" containerID="997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf" Nov 03 13:59:39.582751 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:39.582720 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf\": container with ID starting with 997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf not found: ID does not exist" containerID="997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf" Nov 03 13:59:39.582878 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:39.582767 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf"} err="failed to get container status \"997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf\": rpc error: code = NotFound desc = could not find container \"997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf\": container with ID starting with 997a4408900708973748d535fc6a8a3f8e8414de98c028a033b0b8eacc1d1faf not found: ID does not exist" Nov 03 13:59:39.589940 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:39.589854 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-082668\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:39.638013 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:39.637968 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-082668\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:39.685868 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:39.685824 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-082668\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:40.787850 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:40.787817 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:40.951874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:40.951831 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96dfab60-4888-41a2-a823-2cbd19d153ea" path="/var/lib/kubelet/pods/96dfab60-4888-41a2-a823-2cbd19d153ea/volumes" Nov 03 13:59:41.133406 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:41.133305 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd"] Nov 03 13:59:41.159894 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:41.159860 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:41.526486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:41.526443 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:41.844352 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:41.844324 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:41.855151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:41.851651 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:41.902219 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:41.902184 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593060c0-d51a-43a0-993a-a3bae45ef7a3-kube-api-access-k8pb9\" (UniqueName: \"kubernetes.io/projected/593060c0-d51a-43a0-993a-a3bae45ef7a3-kube-api-access-k8pb9\") pod \"593060c0-d51a-43a0-993a-a3bae45ef7a3\" (UID: \"593060c0-d51a-43a0-993a-a3bae45ef7a3\") " Nov 03 13:59:41.904503 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:41.904457 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593060c0-d51a-43a0-993a-a3bae45ef7a3-kube-api-access-k8pb9" pod "593060c0-d51a-43a0-993a-a3bae45ef7a3" (UID: "593060c0-d51a-43a0-993a-a3bae45ef7a3"). InnerVolumeSpecName "kube-api-access-k8pb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:42.003661 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:42.003624 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8pb9\" (UniqueName: \"kubernetes.io/projected/593060c0-d51a-43a0-993a-a3bae45ef7a3-kube-api-access-k8pb9\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:42.169398 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:42.169358 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:42.568775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:42.568681 4924 generic.go:356] "Generic (PLEG): container finished" podID="593060c0-d51a-43a0-993a-a3bae45ef7a3" containerID="2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116" exitCode=137 Nov 03 13:59:42.568775 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:42.568729 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:42.569039 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:42.568782 4924 scope.go:122] "RemoveContainer" containerID="2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116" Nov 03 13:59:42.579520 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:42.579501 4924 scope.go:122] "RemoveContainer" containerID="2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116" Nov 03 13:59:42.580663 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:42.580615 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116\": container with ID starting with 2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116 not found: ID does not exist" containerID="2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116" Nov 03 13:59:42.580795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:42.580656 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116"} err="failed to get container status \"2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116\": rpc error: code = NotFound desc = could not find container \"2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116\": container with ID starting with 2b3db2a1c5b480a029330a0ec3d1823d456e3d337b7cfb29bb63e08529471116 not found: ID does not exist" Nov 03 13:59:42.628710 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:42.628663 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-082668\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" pod="opendatahub/test-curl-7600-082668" Nov 03 13:59:42.951797 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:42.951768 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" path="/var/lib/kubelet/pods/593060c0-d51a-43a0-993a-a3bae45ef7a3/volumes" Nov 03 13:59:45.899780 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:45.899746 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:46.087676 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:46.087642 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:46.229493 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:46.229451 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:47.647690 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:47.647645 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:48.560909 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:48.560867 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 13:59:49.212605 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:49.212571 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:49.351232 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:49.351193 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:51.324307 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:51.324270 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:51.489335 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:51.489304 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:52.984628 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:52.984591 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:56.157456 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.157392 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" podUID="24b4e1c2-98cf-489d-adb3-9abd680eb85b" containerName="main" containerID="cri-o://61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e" gracePeriod=45 Nov 03 13:59:56.249766 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.249731 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7594-037520"] Nov 03 13:59:56.250297 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.250282 4924 cpu_manager.go:414] "containerMap: removing container" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" containerName="test-curl-7600-082668" Nov 03 13:59:56.250349 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.250298 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" containerName="test-curl-7600-082668" Nov 03 13:59:56.250396 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.250387 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="593060c0-d51a-43a0-993a-a3bae45ef7a3" containerName="test-curl-7600-082668" Nov 03 13:59:56.253709 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.253688 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7594-037520" Nov 03 13:59:56.274346 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.274319 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/79b7e734-933b-40b5-b372-5c55658eefcd-kube-api-access-zbg72\") pod \"test-curl-7594-037520\" (UID: \"79b7e734-933b-40b5-b372-5c55658eefcd\") " pod="opendatahub/test-curl-7594-037520" Nov 03 13:59:56.306869 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.306837 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7594-037520"] Nov 03 13:59:56.349139 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.349111 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kmbtk\"" Nov 03 13:59:56.375459 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.375377 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/79b7e734-933b-40b5-b372-5c55658eefcd-kube-api-access-zbg72\") pod \"test-curl-7594-037520\" (UID: \"79b7e734-933b-40b5-b372-5c55658eefcd\") " pod="opendatahub/test-curl-7594-037520" Nov 03 13:59:56.430610 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.430582 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/79b7e734-933b-40b5-b372-5c55658eefcd-kube-api-access-zbg72\") pod \"test-curl-7594-037520\" (UID: \"79b7e734-933b-40b5-b372-5c55658eefcd\") " pod="opendatahub/test-curl-7594-037520" Nov 03 13:59:56.454571 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.454544 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:59:56.475972 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.475935 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tmp-dir\") pod \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " Nov 03 13:59:56.476173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.476000 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tls-certs\" (UniqueName: \"kubernetes.io/secret/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tls-certs\") pod \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " Nov 03 13:59:56.476173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.476153 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b4e1c2-98cf-489d-adb3-9abd680eb85b-kube-api-access-qhk4r\" (UniqueName: \"kubernetes.io/projected/24b4e1c2-98cf-489d-adb3-9abd680eb85b-kube-api-access-qhk4r\") pod \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " Nov 03 13:59:56.476298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.476244 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-dshm\") pod \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " Nov 03 13:59:56.476298 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.476289 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-model-cache\") pod \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " Nov 03 13:59:56.476963 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.476903 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-model-cache" pod "24b4e1c2-98cf-489d-adb3-9abd680eb85b" (UID: "24b4e1c2-98cf-489d-adb3-9abd680eb85b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:56.477386 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.477354 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tmp-dir" pod "24b4e1c2-98cf-489d-adb3-9abd680eb85b" (UID: "24b4e1c2-98cf-489d-adb3-9abd680eb85b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:56.479489 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.479456 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b4e1c2-98cf-489d-adb3-9abd680eb85b-kube-api-access-qhk4r" pod "24b4e1c2-98cf-489d-adb3-9abd680eb85b" (UID: "24b4e1c2-98cf-489d-adb3-9abd680eb85b"). InnerVolumeSpecName "kube-api-access-qhk4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:56.479874 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.479849 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tls-certs" pod "24b4e1c2-98cf-489d-adb3-9abd680eb85b" (UID: "24b4e1c2-98cf-489d-adb3-9abd680eb85b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:59:56.481440 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.481413 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-dshm" pod "24b4e1c2-98cf-489d-adb3-9abd680eb85b" (UID: "24b4e1c2-98cf-489d-adb3-9abd680eb85b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:56.567022 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.566976 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7594-037520" Nov 03 13:59:56.577603 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.577570 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-home\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-home\") pod \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\" (UID: \"24b4e1c2-98cf-489d-adb3-9abd680eb85b\") " Nov 03 13:59:56.577873 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.577848 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-home" pod "24b4e1c2-98cf-489d-adb3-9abd680eb85b" (UID: "24b4e1c2-98cf-489d-adb3-9abd680eb85b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:56.578095 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.578065 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:56.578182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.578100 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:56.578182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.578115 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b4e1c2-98cf-489d-adb3-9abd680eb85b-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:56.578182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.578129 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhk4r\" (UniqueName: \"kubernetes.io/projected/24b4e1c2-98cf-489d-adb3-9abd680eb85b-kube-api-access-qhk4r\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:56.578182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.578143 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:56.578182 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.578156 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b4e1c2-98cf-489d-adb3-9abd680eb85b-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:56.680735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.680651 4924 generic.go:356] "Generic (PLEG): container finished" podID="24b4e1c2-98cf-489d-adb3-9abd680eb85b" containerID="61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e" exitCode=0 Nov 03 13:59:56.680735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.680690 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" Nov 03 13:59:56.680735 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.680708 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" event={"ID":"24b4e1c2-98cf-489d-adb3-9abd680eb85b","Type":"ContainerDied","Data":"61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e"} Nov 03 13:59:56.680984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.680746 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" event={"ID":"24b4e1c2-98cf-489d-adb3-9abd680eb85b","Type":"ContainerDied","Data":"d03111257a4548f20223ae7daed7deddbedac300289d4e8b5dea1205097f1740"} Nov 03 13:59:56.680984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.680769 4924 scope.go:122] "RemoveContainer" containerID="61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e" Nov 03 13:59:56.693173 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.693148 4924 scope.go:122] "RemoveContainer" containerID="61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e" Nov 03 13:59:56.693431 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:56.693406 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e\": container with ID starting with 61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e not found: ID does not exist" containerID="61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e" Nov 03 13:59:56.693513 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.693437 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e"} err="failed to get container status \"61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e\": rpc error: code = NotFound desc = could not find container \"61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e\": container with ID starting with 61abf96c8434858b8fff5d464464ed155e6b4a92e59a09e311ec4212eb361e9e not found: ID does not exist" Nov 03 13:59:56.836172 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.836139 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd"] Nov 03 13:59:56.840292 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.840262 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd"] Nov 03 13:59:56.939101 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.938998 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7594-037520"] Nov 03 13:59:56.950885 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:59:56.950835 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79b7e734_933b_40b5_b372_5c55658eefcd.slice/crio-16989d947f914a51e4847ad22a1676ff4b7b9c1326c58f8dc4722c8dc75315ba WatchSource:0}: Error finding container 16989d947f914a51e4847ad22a1676ff4b7b9c1326c58f8dc4722c8dc75315ba: Status 404 returned error can't find the container with id 16989d947f914a51e4847ad22a1676ff4b7b9c1326c58f8dc4722c8dc75315ba Nov 03 13:59:56.953151 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:56.953129 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b4e1c2-98cf-489d-adb3-9abd680eb85b" path="/var/lib/kubelet/pods/24b4e1c2-98cf-489d-adb3-9abd680eb85b/volumes" Nov 03 13:59:57.210000 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.209910 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw"] Nov 03 13:59:57.210494 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.210127 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" containerID="cri-o://afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5" gracePeriod=30 Nov 03 13:59:57.345730 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.345688 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-d341af/auth-test-model-d341af-kserve-56cb74fc85-n4drd" podUID="24b4e1c2-98cf-489d-adb3-9abd680eb85b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.125:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:59:57.691227 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.691185 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7594-037520" event={"ID":"79b7e734-933b-40b5-b372-5c55658eefcd","Type":"ContainerStarted","Data":"b0bf36125cb8928b3027f794b9de8f4f144ad4488524c3617b3180e9b0276953"} Nov 03 13:59:57.691421 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.691238 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7594-037520" event={"ID":"79b7e734-933b-40b5-b372-5c55658eefcd","Type":"ContainerStarted","Data":"16989d947f914a51e4847ad22a1676ff4b7b9c1326c58f8dc4722c8dc75315ba"} Nov 03 13:59:57.724925 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.724891 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79"] Nov 03 13:59:57.725696 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.725673 4924 cpu_manager.go:414] "containerMap: removing container" podUID="24b4e1c2-98cf-489d-adb3-9abd680eb85b" containerName="main" Nov 03 13:59:57.725800 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.725698 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b4e1c2-98cf-489d-adb3-9abd680eb85b" containerName="main" Nov 03 13:59:57.725867 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.725850 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b4e1c2-98cf-489d-adb3-9abd680eb85b" containerName="main" Nov 03 13:59:57.729776 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.729754 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.781057 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.780562 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-efbdec\"/\"default-dockercfg-zmtmd\"" Nov 03 13:59:57.782148 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.782124 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-efbdec\"/\"auth-test-model-efbdec-kserve-self-signed-certs\"" Nov 03 13:59:57.782323 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.782310 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-efbdec\"/\"kube-root-ca.crt\"" Nov 03 13:59:57.782380 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.782343 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-efbdec\"/\"openshift-service-ca.crt\"" Nov 03 13:59:57.784401 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.784377 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79"] Nov 03 13:59:57.790755 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.790729 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjgh\" (UniqueName: \"kubernetes.io/projected/206eeab5-6be2-4392-b95d-4961a5aa1d43-kube-api-access-2wjgh\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.790858 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.790784 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-model-cache\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.790911 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.790898 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-home\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.790952 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.790932 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-dshm\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.791119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.791100 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-tmp-dir\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.791177 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.791162 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206eeab5-6be2-4392-b95d-4961a5aa1d43-tls-certs\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.797650 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.796759 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7594-037520" podStartSLOduration=1.796745956 podStartE2EDuration="1.796745956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:57.79658556 +0000 UTC m=+3330.990989157" watchObservedRunningTime="2026-11-03 13:59:57.796745956 +0000 UTC m=+3330.991149551" Nov 03 13:59:57.804678 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.804658 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:57.892412 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.892372 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjgh\" (UniqueName: \"kubernetes.io/projected/206eeab5-6be2-4392-b95d-4961a5aa1d43-kube-api-access-2wjgh\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.892584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.892437 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-model-cache\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.892584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.892523 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-home\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.892584 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.892563 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-dshm\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.892732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.892635 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-tmp-dir\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.892732 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.892685 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206eeab5-6be2-4392-b95d-4961a5aa1d43-tls-certs\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.893127 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.893100 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-home\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.893984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.893419 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-model-cache\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.893984 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.893626 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-tmp-dir\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.896707 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.896507 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-dshm\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.896795 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.896683 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206eeab5-6be2-4392-b95d-4961a5aa1d43-tls-certs\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:57.953785 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:57.953706 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjgh\" (UniqueName: \"kubernetes.io/projected/206eeab5-6be2-4392-b95d-4961a5aa1d43-kube-api-access-2wjgh\") pod \"auth-test-model-efbdec-kserve-55589d4897-d8n79\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:58.042752 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:58.042716 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:58.139640 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:58.139605 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 13:59:58.419104 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:58.419041 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79"] Nov 03 13:59:58.420547 ip-10-0-129-63 kubenswrapper[4924]: W1103 13:59:58.420516 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod206eeab5_6be2_4392_b95d_4961a5aa1d43.slice/crio-0a3330d68cfbd8bcf1c6d3bb13f505ece789fe4b0e0ec28ae9122a7de254101b WatchSource:0}: Error finding container 0a3330d68cfbd8bcf1c6d3bb13f505ece789fe4b0e0ec28ae9122a7de254101b: Status 404 returned error can't find the container with id 0a3330d68cfbd8bcf1c6d3bb13f505ece789fe4b0e0ec28ae9122a7de254101b Nov 03 13:59:58.699815 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:58.699729 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" event={"ID":"206eeab5-6be2-4392-b95d-4961a5aa1d43","Type":"ContainerStarted","Data":"ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5"} Nov 03 13:59:58.699815 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:58.699779 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" event={"ID":"206eeab5-6be2-4392-b95d-4961a5aa1d43","Type":"ContainerStarted","Data":"0a3330d68cfbd8bcf1c6d3bb13f505ece789fe4b0e0ec28ae9122a7de254101b"} Nov 03 13:59:58.819119 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:58.819053 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" podStartSLOduration=1.819039876 podStartE2EDuration="1.819039876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:58.817694235 +0000 UTC m=+3332.012097830" watchObservedRunningTime="2026-11-03 13:59:58.819039876 +0000 UTC m=+3332.013443632" Nov 03 13:59:59.043652 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:59.043572 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 13:59:59.375486 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:59.374998 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7594-037520"] Nov 03 13:59:59.385388 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:59.385352 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7594-037520"] Nov 03 13:59:59.708316 ip-10-0-129-63 kubenswrapper[4924]: I1103 13:59:59.708244 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7594-037520" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" containerName="test-curl-7594-037520" containerID="cri-o://b0bf36125cb8928b3027f794b9de8f4f144ad4488524c3617b3180e9b0276953" gracePeriod=2 Nov 03 13:59:59.756405 ip-10-0-129-63 kubenswrapper[4924]: E1103 13:59:59.756365 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7594-037520\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" pod="opendatahub/test-curl-7594-037520" Nov 03 14:00:00.167480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.167438 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p"] Nov 03 14:00:00.167973 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.167958 4924 cpu_manager.go:414] "containerMap: removing container" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" containerName="test-curl-7594-037520" Nov 03 14:00:00.168042 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.167973 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" containerName="test-curl-7594-037520" Nov 03 14:00:00.168134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.168105 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" containerName="test-curl-7594-037520" Nov 03 14:00:00.171531 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.171508 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq"] Nov 03 14:00:00.171669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.171637 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:00.175529 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.175506 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh"] Nov 03 14:00:00.175671 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.175634 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:00.179322 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.179298 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:00.202277 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.202245 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664"] Nov 03 14:00:00.206413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.206390 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:00.218371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.218343 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5rp\" (UniqueName: \"kubernetes.io/projected/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-kube-api-access-2b5rp\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq\" (UID: \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:00.218522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.218382 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f25c6-1af8-400f-ade0-c41097146ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664\" (UID: \"fd8f25c6-1af8-400f-ade0-c41097146ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:00.218522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.218473 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq\" (UID: \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:00.218522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.218504 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wnm\" (UniqueName: \"kubernetes.io/projected/e92ba637-ef5f-4ec7-852f-0966c9357d2d-kube-api-access-79wnm\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh\" (UID: \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:00.218693 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.218553 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4bz\" (UniqueName: \"kubernetes.io/projected/fd8f25c6-1af8-400f-ade0-c41097146ab2-kube-api-access-fq4bz\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664\" (UID: \"fd8f25c6-1af8-400f-ade0-c41097146ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:00.218693 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.218645 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba637-ef5f-4ec7-852f-0966c9357d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh\" (UID: \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:00.218693 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.218666 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33af9ebf-6ee4-49be-ae45-1eb279067c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p\" (UID: \"33af9ebf-6ee4-49be-ae45-1eb279067c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:00.218849 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.218761 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kksc\" (UniqueName: \"kubernetes.io/projected/33af9ebf-6ee4-49be-ae45-1eb279067c83-kube-api-access-5kksc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p\" (UID: \"33af9ebf-6ee4-49be-ae45-1eb279067c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:00.220296 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.220273 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7"] Nov 03 14:00:00.223969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.223948 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:00.225348 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.225327 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:00:00.227227 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.227205 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568"] Nov 03 14:00:00.231796 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.231776 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:00.261595 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.261563 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq"] Nov 03 14:00:00.264669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.264641 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p"] Nov 03 14:00:00.315664 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.315620 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664"] Nov 03 14:00:00.318817 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.318779 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7"] Nov 03 14:00:00.320456 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320423 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5rp\" (UniqueName: \"kubernetes.io/projected/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-kube-api-access-2b5rp\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq\" (UID: \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:00.320598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320477 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f25c6-1af8-400f-ade0-c41097146ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664\" (UID: \"fd8f25c6-1af8-400f-ade0-c41097146ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:00.320598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320531 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq\" (UID: \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:00.320598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320556 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wnm\" (UniqueName: \"kubernetes.io/projected/e92ba637-ef5f-4ec7-852f-0966c9357d2d-kube-api-access-79wnm\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh\" (UID: \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:00.320598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320595 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4bz\" (UniqueName: \"kubernetes.io/projected/fd8f25c6-1af8-400f-ade0-c41097146ab2-kube-api-access-fq4bz\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664\" (UID: \"fd8f25c6-1af8-400f-ade0-c41097146ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:00.320776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320623 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e504b-75b2-4266-88a1-4badcd1407da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895240-9vcl7\" (UID: \"7e3e504b-75b2-4266-88a1-4badcd1407da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:00.320776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320681 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba637-ef5f-4ec7-852f-0966c9357d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh\" (UID: \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:00.320776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320702 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33af9ebf-6ee4-49be-ae45-1eb279067c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p\" (UID: \"33af9ebf-6ee4-49be-ae45-1eb279067c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:00.320776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320734 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b57hr\" (UniqueName: \"kubernetes.io/projected/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-kube-api-access-b57hr\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568\" (UID: \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:00.320776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320757 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8v4\" (UniqueName: \"kubernetes.io/projected/7e3e504b-75b2-4266-88a1-4badcd1407da-kube-api-access-8l8v4\") pod \"maas-api-key-cleanup-29895240-9vcl7\" (UID: \"7e3e504b-75b2-4266-88a1-4badcd1407da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:00.320941 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320805 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568\" (UID: \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:00.320941 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.320841 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kksc\" (UniqueName: \"kubernetes.io/projected/33af9ebf-6ee4-49be-ae45-1eb279067c83-kube-api-access-5kksc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p\" (UID: \"33af9ebf-6ee4-49be-ae45-1eb279067c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:00.321933 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.321904 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba637-ef5f-4ec7-852f-0966c9357d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh\" (UID: \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:00.321933 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.321910 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq\" (UID: \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:00.322533 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.322510 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f25c6-1af8-400f-ade0-c41097146ab2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664\" (UID: \"fd8f25c6-1af8-400f-ade0-c41097146ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:00.322650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.322628 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33af9ebf-6ee4-49be-ae45-1eb279067c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p\" (UID: \"33af9ebf-6ee4-49be-ae45-1eb279067c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:00.327335 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:00.327310 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7594-037520\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" pod="opendatahub/test-curl-7594-037520" Nov 03 14:00:00.388093 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:00.382512 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7594-037520\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" pod="opendatahub/test-curl-7594-037520" Nov 03 14:00:00.388093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.386192 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5rp\" (UniqueName: \"kubernetes.io/projected/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-kube-api-access-2b5rp\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq\" (UID: \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:00.396386 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.394102 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4bz\" (UniqueName: \"kubernetes.io/projected/fd8f25c6-1af8-400f-ade0-c41097146ab2-kube-api-access-fq4bz\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664\" (UID: \"fd8f25c6-1af8-400f-ade0-c41097146ab2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:00.397863 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.397829 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kksc\" (UniqueName: \"kubernetes.io/projected/33af9ebf-6ee4-49be-ae45-1eb279067c83-kube-api-access-5kksc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p\" (UID: \"33af9ebf-6ee4-49be-ae45-1eb279067c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:00.404118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.399445 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wnm\" (UniqueName: \"kubernetes.io/projected/e92ba637-ef5f-4ec7-852f-0966c9357d2d-kube-api-access-79wnm\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh\" (UID: \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:00.416300 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.416269 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh"] Nov 03 14:00:00.421105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.421016 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568"] Nov 03 14:00:00.421778 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.421750 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b57hr\" (UniqueName: \"kubernetes.io/projected/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-kube-api-access-b57hr\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568\" (UID: \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:00.421910 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.421895 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8v4\" (UniqueName: \"kubernetes.io/projected/7e3e504b-75b2-4266-88a1-4badcd1407da-kube-api-access-8l8v4\") pod \"maas-api-key-cleanup-29895240-9vcl7\" (UID: \"7e3e504b-75b2-4266-88a1-4badcd1407da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:00.422040 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.422027 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568\" (UID: \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:00.422267 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.422251 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e504b-75b2-4266-88a1-4badcd1407da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895240-9vcl7\" (UID: \"7e3e504b-75b2-4266-88a1-4badcd1407da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:00.423174 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.423143 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e504b-75b2-4266-88a1-4badcd1407da-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895240-9vcl7\" (UID: \"7e3e504b-75b2-4266-88a1-4badcd1407da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:00.423499 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.423476 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568\" (UID: \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:00.478487 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.478460 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8v4\" (UniqueName: \"kubernetes.io/projected/7e3e504b-75b2-4266-88a1-4badcd1407da-kube-api-access-8l8v4\") pod \"maas-api-key-cleanup-29895240-9vcl7\" (UID: \"7e3e504b-75b2-4266-88a1-4badcd1407da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:00.478676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.478543 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b57hr\" (UniqueName: \"kubernetes.io/projected/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-kube-api-access-b57hr\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568\" (UID: \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:00.487392 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.487367 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:00.494916 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.494895 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:00.503498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.503475 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:00.519665 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.519636 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:00.537588 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.537551 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:00.545345 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.545317 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:00.899343 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:00:00.899313 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33af9ebf_6ee4_49be_ae45_1eb279067c83.slice/crio-d8fe83a10839f9cf86eafdf73791cd68b4f82fd1a378cda3dfb88364190a6f92 WatchSource:0}: Error finding container d8fe83a10839f9cf86eafdf73791cd68b4f82fd1a378cda3dfb88364190a6f92: Status 404 returned error can't find the container with id d8fe83a10839f9cf86eafdf73791cd68b4f82fd1a378cda3dfb88364190a6f92 Nov 03 14:00:00.900016 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.899989 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p"] Nov 03 14:00:00.936570 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.936510 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq"] Nov 03 14:00:00.944108 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:00:00.944059 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a187191_fbd6_43b3_b9fc_b9c136b9d8fe.slice/crio-28a32d3a800aae281619feea8e4810d274802dc6f5c697d44ac8bb9b774c2f04 WatchSource:0}: Error finding container 28a32d3a800aae281619feea8e4810d274802dc6f5c697d44ac8bb9b774c2f04: Status 404 returned error can't find the container with id 28a32d3a800aae281619feea8e4810d274802dc6f5c697d44ac8bb9b774c2f04 Nov 03 14:00:00.949713 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:00:00.949685 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8f25c6_1af8_400f_ade0_c41097146ab2.slice/crio-4a52317fb3ec6e16dbbd82dda78acc07ac62120ba3cf8bbcf7fd805ead0815b4 WatchSource:0}: Error finding container 4a52317fb3ec6e16dbbd82dda78acc07ac62120ba3cf8bbcf7fd805ead0815b4: Status 404 returned error can't find the container with id 4a52317fb3ec6e16dbbd82dda78acc07ac62120ba3cf8bbcf7fd805ead0815b4 Nov 03 14:00:00.954215 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:00.954188 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664"] Nov 03 14:00:01.011390 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.011354 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7"] Nov 03 14:00:01.016141 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:00:01.016113 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3e504b_75b2_4266_88a1_4badcd1407da.slice/crio-d06198b57f8a81fa1b42ae8b9c30ab0af7fb360dd06b4dc2e5dc8618fcbc27bb WatchSource:0}: Error finding container d06198b57f8a81fa1b42ae8b9c30ab0af7fb360dd06b4dc2e5dc8618fcbc27bb: Status 404 returned error can't find the container with id d06198b57f8a81fa1b42ae8b9c30ab0af7fb360dd06b4dc2e5dc8618fcbc27bb Nov 03 14:00:01.035375 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:00:01.035336 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538839bf_d1eb_4d7c_a5f1_17e1d28aec78.slice/crio-7b65979a91213cf19d7b1a4ea3c4720fbc1d03e04ea2a4ce6577a8df4fa8b07e WatchSource:0}: Error finding container 7b65979a91213cf19d7b1a4ea3c4720fbc1d03e04ea2a4ce6577a8df4fa8b07e: Status 404 returned error can't find the container with id 7b65979a91213cf19d7b1a4ea3c4720fbc1d03e04ea2a4ce6577a8df4fa8b07e Nov 03 14:00:01.036194 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.036169 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568"] Nov 03 14:00:01.043729 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.043651 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:01.066546 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.066522 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh"] Nov 03 14:00:01.068695 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:00:01.068662 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92ba637_ef5f_4ec7_852f_0966c9357d2d.slice/crio-910f1299f43413ab1c956b81c54000a3367c7a18286655f294cc02444bbd9d1f WatchSource:0}: Error finding container 910f1299f43413ab1c956b81c54000a3367c7a18286655f294cc02444bbd9d1f: Status 404 returned error can't find the container with id 910f1299f43413ab1c956b81c54000a3367c7a18286655f294cc02444bbd9d1f Nov 03 14:00:01.425558 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.425522 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:01.739038 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.739005 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" event={"ID":"33af9ebf-6ee4-49be-ae45-1eb279067c83","Type":"ContainerStarted","Data":"55cd503ebb77280d371be5665f12aece596ec9410bfce9dd0a84fb9d7ce03449"} Nov 03 14:00:01.739208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.739115 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" event={"ID":"33af9ebf-6ee4-49be-ae45-1eb279067c83","Type":"ContainerStarted","Data":"d8fe83a10839f9cf86eafdf73791cd68b4f82fd1a378cda3dfb88364190a6f92"} Nov 03 14:00:01.740937 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.740903 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" event={"ID":"538839bf-d1eb-4d7c-a5f1-17e1d28aec78","Type":"ContainerStarted","Data":"c7a7076cf80c05df33cfac8e4bf94fc376087bd4cc4e3ae220383cef91a9edd3"} Nov 03 14:00:01.741040 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.740954 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" event={"ID":"538839bf-d1eb-4d7c-a5f1-17e1d28aec78","Type":"ContainerStarted","Data":"7b65979a91213cf19d7b1a4ea3c4720fbc1d03e04ea2a4ce6577a8df4fa8b07e"} Nov 03 14:00:01.742650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.742625 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" event={"ID":"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe","Type":"ContainerStarted","Data":"dac77820f50a3ccfaa68f9c353267295950d7f7f10ca70c033e41a27c8f00948"} Nov 03 14:00:01.742787 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.742672 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" event={"ID":"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe","Type":"ContainerStarted","Data":"28a32d3a800aae281619feea8e4810d274802dc6f5c697d44ac8bb9b774c2f04"} Nov 03 14:00:01.744429 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.744327 4924 generic.go:356] "Generic (PLEG): container finished" podID="79b7e734-933b-40b5-b372-5c55658eefcd" containerID="b0bf36125cb8928b3027f794b9de8f4f144ad4488524c3617b3180e9b0276953" exitCode=137 Nov 03 14:00:01.746202 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.746172 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" event={"ID":"7e3e504b-75b2-4266-88a1-4badcd1407da","Type":"ContainerStarted","Data":"b4b2899633695361704f8bcbd6c31d0cc79a46d3cc4aa0b3b5d56c61a69606dc"} Nov 03 14:00:01.746300 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.746217 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" event={"ID":"7e3e504b-75b2-4266-88a1-4badcd1407da","Type":"ContainerStarted","Data":"d06198b57f8a81fa1b42ae8b9c30ab0af7fb360dd06b4dc2e5dc8618fcbc27bb"} Nov 03 14:00:01.748145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.748121 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" event={"ID":"e92ba637-ef5f-4ec7-852f-0966c9357d2d","Type":"ContainerStarted","Data":"ffb6049c4efca9dde29bb51845a07be3325b75d15c458cff3a12855b14382e80"} Nov 03 14:00:01.748248 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.748164 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" event={"ID":"e92ba637-ef5f-4ec7-852f-0966c9357d2d","Type":"ContainerStarted","Data":"910f1299f43413ab1c956b81c54000a3367c7a18286655f294cc02444bbd9d1f"} Nov 03 14:00:01.749902 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.749879 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" event={"ID":"fd8f25c6-1af8-400f-ade0-c41097146ab2","Type":"ContainerStarted","Data":"67462e748ac0669a1df38cead91af67f155536bab43d3488d039074fcfb2e8b6"} Nov 03 14:00:01.749986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.749918 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" event={"ID":"fd8f25c6-1af8-400f-ade0-c41097146ab2","Type":"ContainerStarted","Data":"4a52317fb3ec6e16dbbd82dda78acc07ac62120ba3cf8bbcf7fd805ead0815b4"} Nov 03 14:00:01.857587 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.857444 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" podStartSLOduration=1.857423454 podStartE2EDuration="1.857423454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:01.854505475 +0000 UTC m=+3335.048909057" watchObservedRunningTime="2026-11-03 14:00:01.857423454 +0000 UTC m=+3335.051827049" Nov 03 14:00:01.866924 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.866890 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:01.968947 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.968891 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" podStartSLOduration=1.968873748 podStartE2EDuration="1.968873748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:01.965401614 +0000 UTC m=+3335.159805209" watchObservedRunningTime="2026-11-03 14:00:01.968873748 +0000 UTC m=+3335.163277366" Nov 03 14:00:01.995198 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:01.995164 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7594-037520" Nov 03 14:00:02.024319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.024281 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:02.077914 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.077847 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" podStartSLOduration=2.077833459 podStartE2EDuration="2.077833459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:02.075799519 +0000 UTC m=+3335.270203114" watchObservedRunningTime="2026-11-03 14:00:02.077833459 +0000 UTC m=+3335.272237054" Nov 03 14:00:02.098592 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:02.098546 4924 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92ba637_ef5f_4ec7_852f_0966c9357d2d.slice/crio-conmon-ffb6049c4efca9dde29bb51845a07be3325b75d15c458cff3a12855b14382e80.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33af9ebf_6ee4_49be_ae45_1eb279067c83.slice/crio-conmon-55cd503ebb77280d371be5665f12aece596ec9410bfce9dd0a84fb9d7ce03449.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33af9ebf_6ee4_49be_ae45_1eb279067c83.slice/crio-55cd503ebb77280d371be5665f12aece596ec9410bfce9dd0a84fb9d7ce03449.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92ba637_ef5f_4ec7_852f_0966c9357d2d.slice/crio-ffb6049c4efca9dde29bb51845a07be3325b75d15c458cff3a12855b14382e80.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:00:02.149113 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.149063 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b7e734-933b-40b5-b372-5c55658eefcd-kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/79b7e734-933b-40b5-b372-5c55658eefcd-kube-api-access-zbg72\") pod \"79b7e734-933b-40b5-b372-5c55658eefcd\" (UID: \"79b7e734-933b-40b5-b372-5c55658eefcd\") " Nov 03 14:00:02.151037 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.151007 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b7e734-933b-40b5-b372-5c55658eefcd-kube-api-access-zbg72" pod "79b7e734-933b-40b5-b372-5c55658eefcd" (UID: "79b7e734-933b-40b5-b372-5c55658eefcd"). InnerVolumeSpecName "kube-api-access-zbg72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:02.188926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.188862 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" podStartSLOduration=2.188844404 podStartE2EDuration="2.188844404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:02.186822337 +0000 UTC m=+3335.381225932" watchObservedRunningTime="2026-11-03 14:00:02.188844404 +0000 UTC m=+3335.383248002" Nov 03 14:00:02.267338 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.266864 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbg72\" (UniqueName: \"kubernetes.io/projected/79b7e734-933b-40b5-b372-5c55658eefcd-kube-api-access-zbg72\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.297236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.297170 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" podStartSLOduration=2.297155588 podStartE2EDuration="2.297155588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:02.293824533 +0000 UTC m=+3335.488228130" watchObservedRunningTime="2026-11-03 14:00:02.297155588 +0000 UTC m=+3335.491559227" Nov 03 14:00:02.400876 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.400822 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" podStartSLOduration=2.400807695 podStartE2EDuration="2.400807695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:02.400400857 +0000 UTC m=+3335.594804451" watchObservedRunningTime="2026-11-03 14:00:02.400807695 +0000 UTC m=+3335.595211306" Nov 03 14:00:02.532000 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.531969 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 14:00:02.571233 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.571184 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-kube-api-access-87dqm\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-kube-api-access-87dqm\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.571422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.571276 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istiod-ca-cert\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.571422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.571327 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-token\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-token\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.571422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.571373 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-data\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.571723 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.571693 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istiod-ca-cert" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:02.571891 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.571870 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-podinfo\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.572171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.572156 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-socket\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.572289 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.572048 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-data" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:02.572405 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.572390 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-certs\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.572526 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.572514 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-envoy\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.572607 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.572565 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-socket" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:02.572668 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.572656 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-certs" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:02.572761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.572748 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-credential-socket\") pod \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\" (UID: \"d70a0384-eb61-46e6-8b0d-0105f34de2e3\") " Nov 03 14:00:02.573222 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.573194 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-credential-socket" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:02.573392 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.573378 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.573504 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.573493 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.573585 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.573574 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.573662 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.573652 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.574721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.574689 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-podinfo" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:00:02.574977 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.574949 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-envoy" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:02.575061 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.574954 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-kube-api-access-87dqm" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "kube-api-access-87dqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:02.575149 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.575132 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-token" pod "d70a0384-eb61-46e6-8b0d-0105f34de2e3" (UID: "d70a0384-eb61-46e6-8b0d-0105f34de2e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:02.676842 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.676811 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87dqm\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-kube-api-access-87dqm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.677046 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.677032 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.677172 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.677158 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.677251 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.677241 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.677335 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.677325 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70a0384-eb61-46e6-8b0d-0105f34de2e3-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:02.759919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.758354 4924 generic.go:356] "Generic (PLEG): container finished" podID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerID="afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5" exitCode=0 Nov 03 14:00:02.759919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.758462 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" event={"ID":"d70a0384-eb61-46e6-8b0d-0105f34de2e3","Type":"ContainerDied","Data":"afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5"} Nov 03 14:00:02.759919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.758525 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" event={"ID":"d70a0384-eb61-46e6-8b0d-0105f34de2e3","Type":"ContainerDied","Data":"333785298e78bb742d17fa391ad23cdf5486e99fc63fc71f58dfb8713e9f7ad4"} Nov 03 14:00:02.759919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.758561 4924 scope.go:122] "RemoveContainer" containerID="afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5" Nov 03 14:00:02.759919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.758772 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw" Nov 03 14:00:02.764146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.764110 4924 generic.go:356] "Generic (PLEG): container finished" podID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerID="ffb6049c4efca9dde29bb51845a07be3325b75d15c458cff3a12855b14382e80" exitCode=60 Nov 03 14:00:02.764281 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.764217 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" event={"ID":"e92ba637-ef5f-4ec7-852f-0966c9357d2d","Type":"ContainerDied","Data":"ffb6049c4efca9dde29bb51845a07be3325b75d15c458cff3a12855b14382e80"} Nov 03 14:00:02.764512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.764482 4924 scope.go:122] "RemoveContainer" containerID="ffb6049c4efca9dde29bb51845a07be3325b75d15c458cff3a12855b14382e80" Nov 03 14:00:02.767248 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.767223 4924 generic.go:356] "Generic (PLEG): container finished" podID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerID="67462e748ac0669a1df38cead91af67f155536bab43d3488d039074fcfb2e8b6" exitCode=60 Nov 03 14:00:02.767493 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.767471 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" event={"ID":"fd8f25c6-1af8-400f-ade0-c41097146ab2","Type":"ContainerDied","Data":"67462e748ac0669a1df38cead91af67f155536bab43d3488d039074fcfb2e8b6"} Nov 03 14:00:02.767612 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.767600 4924 scope.go:122] "RemoveContainer" containerID="67462e748ac0669a1df38cead91af67f155536bab43d3488d039074fcfb2e8b6" Nov 03 14:00:02.772437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.772392 4924 generic.go:356] "Generic (PLEG): container finished" podID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerID="55cd503ebb77280d371be5665f12aece596ec9410bfce9dd0a84fb9d7ce03449" exitCode=60 Nov 03 14:00:02.772593 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.772543 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" event={"ID":"33af9ebf-6ee4-49be-ae45-1eb279067c83","Type":"ContainerDied","Data":"55cd503ebb77280d371be5665f12aece596ec9410bfce9dd0a84fb9d7ce03449"} Nov 03 14:00:02.772907 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.772888 4924 scope.go:122] "RemoveContainer" containerID="55cd503ebb77280d371be5665f12aece596ec9410bfce9dd0a84fb9d7ce03449" Nov 03 14:00:02.774402 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.774378 4924 generic.go:356] "Generic (PLEG): container finished" podID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerID="c7a7076cf80c05df33cfac8e4bf94fc376087bd4cc4e3ae220383cef91a9edd3" exitCode=60 Nov 03 14:00:02.774497 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.774457 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" event={"ID":"538839bf-d1eb-4d7c-a5f1-17e1d28aec78","Type":"ContainerDied","Data":"c7a7076cf80c05df33cfac8e4bf94fc376087bd4cc4e3ae220383cef91a9edd3"} Nov 03 14:00:02.774710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.774693 4924 scope.go:122] "RemoveContainer" containerID="c7a7076cf80c05df33cfac8e4bf94fc376087bd4cc4e3ae220383cef91a9edd3" Nov 03 14:00:02.777135 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.777049 4924 generic.go:356] "Generic (PLEG): container finished" podID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerID="dac77820f50a3ccfaa68f9c353267295950d7f7f10ca70c033e41a27c8f00948" exitCode=60 Nov 03 14:00:02.777227 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.777196 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" event={"ID":"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe","Type":"ContainerDied","Data":"dac77820f50a3ccfaa68f9c353267295950d7f7f10ca70c033e41a27c8f00948"} Nov 03 14:00:02.778026 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.778007 4924 scope.go:122] "RemoveContainer" containerID="dac77820f50a3ccfaa68f9c353267295950d7f7f10ca70c033e41a27c8f00948" Nov 03 14:00:02.780266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.780247 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7594-037520" Nov 03 14:00:02.782173 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.782053 4924 generic.go:356] "Generic (PLEG): container finished" podID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerID="b4b2899633695361704f8bcbd6c31d0cc79a46d3cc4aa0b3b5d56c61a69606dc" exitCode=60 Nov 03 14:00:02.782173 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.782101 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" event={"ID":"7e3e504b-75b2-4266-88a1-4badcd1407da","Type":"ContainerDied","Data":"b4b2899633695361704f8bcbd6c31d0cc79a46d3cc4aa0b3b5d56c61a69606dc"} Nov 03 14:00:02.782386 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.782370 4924 scope.go:122] "RemoveContainer" containerID="b4b2899633695361704f8bcbd6c31d0cc79a46d3cc4aa0b3b5d56c61a69606dc" Nov 03 14:00:02.788897 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.788877 4924 scope.go:122] "RemoveContainer" containerID="afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5" Nov 03 14:00:02.789424 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:02.789399 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5\": container with ID starting with afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5 not found: ID does not exist" containerID="afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5" Nov 03 14:00:02.789501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.789440 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5"} err="failed to get container status \"afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5\": rpc error: code = NotFound desc = could not find container \"afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5\": container with ID starting with afba8c8b5bb506f043085a01049aa032747092a36f10537b89c5030f55fa0cd5 not found: ID does not exist" Nov 03 14:00:02.789501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.789463 4924 scope.go:122] "RemoveContainer" containerID="b0bf36125cb8928b3027f794b9de8f4f144ad4488524c3617b3180e9b0276953" Nov 03 14:00:02.952498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:02.952463 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" path="/var/lib/kubelet/pods/79b7e734-933b-40b5-b372-5c55658eefcd/volumes" Nov 03 14:00:03.151548 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.151517 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw"] Nov 03 14:00:03.155455 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.155432 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-m2gxw"] Nov 03 14:00:03.199306 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:03.199220 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7594-037520\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="79b7e734-933b-40b5-b372-5c55658eefcd" pod="opendatahub/test-curl-7594-037520" Nov 03 14:00:03.271891 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.271824 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:03.791628 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.791590 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" event={"ID":"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe","Type":"ContainerStarted","Data":"148219559a906799f599681aed713c1c1e9a3e550f455be57257d0b4c0adbcdb"} Nov 03 14:00:03.794726 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.794699 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" event={"ID":"7e3e504b-75b2-4266-88a1-4badcd1407da","Type":"ContainerStarted","Data":"7dc931c6e129d0a4a02660dffa599d176fee368267ede228d50fa39dd710cf6c"} Nov 03 14:00:03.797947 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.797917 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" event={"ID":"e92ba637-ef5f-4ec7-852f-0966c9357d2d","Type":"ContainerStarted","Data":"ab4840641f7f2c252124477fcbb0fe4a0b93d7a3348c34285357ba1c66cc49ef"} Nov 03 14:00:03.799989 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.799962 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" event={"ID":"fd8f25c6-1af8-400f-ade0-c41097146ab2","Type":"ContainerStarted","Data":"aad5ab9e4763ae5bba019976d01cd0eeca249eb0f6136f123724c06e0a3d51ad"} Nov 03 14:00:03.802127 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.802101 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" event={"ID":"33af9ebf-6ee4-49be-ae45-1eb279067c83","Type":"ContainerStarted","Data":"3561cef25b20db900fa4687d96eb64da94d11d0bd25edd6202e5180b4b423877"} Nov 03 14:00:03.804141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.804113 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" event={"ID":"538839bf-d1eb-4d7c-a5f1-17e1d28aec78","Type":"ContainerStarted","Data":"d0e9c813dfff2e4ab89e2d4538f24dfb4dc89a6a33e52cafbcff95bb8c1443e4"} Nov 03 14:00:03.825323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.825294 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn"] Nov 03 14:00:03.825831 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.825818 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" Nov 03 14:00:03.825870 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.825833 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" Nov 03 14:00:03.825945 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.825936 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" containerName="istio-proxy" Nov 03 14:00:03.832893 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.832861 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.884107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.883387 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-921466\"/\"openshift-service-ca.crt\"" Nov 03 14:00:03.884107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.883551 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-921466\"/\"noten-model-921466-kserve-self-signed-certs\"" Nov 03 14:00:03.884107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.883685 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-921466\"/\"kube-root-ca.crt\"" Nov 03 14:00:03.884107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.883811 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-921466\"/\"default-dockercfg-449ql\"" Nov 03 14:00:03.885144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.885118 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn"] Nov 03 14:00:03.891709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.891679 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-home\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.891811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.891758 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b09327-0a3f-4c6f-be64-211ff018d34f-tls-certs\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.891811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.891785 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-dshm\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.892818 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.892796 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-model-cache\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.892928 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.892877 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpw2d\" (UniqueName: \"kubernetes.io/projected/f0b09327-0a3f-4c6f-be64-211ff018d34f-kube-api-access-qpw2d\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.892986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.892937 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-tmp-dir\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.922314 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.922275 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:03.994487 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.994446 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-home\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.994725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.994516 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b09327-0a3f-4c6f-be64-211ff018d34f-tls-certs\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.994725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.994578 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-dshm\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.994725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.994629 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-model-cache\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.994725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.994663 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpw2d\" (UniqueName: \"kubernetes.io/projected/f0b09327-0a3f-4c6f-be64-211ff018d34f-kube-api-access-qpw2d\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.994725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.994688 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-tmp-dir\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.995552 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.995180 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-tmp-dir\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.995552 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.995532 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-home\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.995965 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.995943 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-model-cache\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.997904 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.997879 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-dshm\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:03.998358 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:03.998333 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b09327-0a3f-4c6f-be64-211ff018d34f-tls-certs\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:04.049547 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.049471 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpw2d\" (UniqueName: \"kubernetes.io/projected/f0b09327-0a3f-4c6f-be64-211ff018d34f-kube-api-access-qpw2d\") pod \"noten-model-921466-kserve-64d75f5cfc-vwwtn\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:04.328571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.328477 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:04.732792 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:00:04.732750 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0b09327_0a3f_4c6f_be64_211ff018d34f.slice/crio-6b036747f4b782683033cb74a27b9d566d39a18e82b7bc0dfbd18c1bec009560 WatchSource:0}: Error finding container 6b036747f4b782683033cb74a27b9d566d39a18e82b7bc0dfbd18c1bec009560: Status 404 returned error can't find the container with id 6b036747f4b782683033cb74a27b9d566d39a18e82b7bc0dfbd18c1bec009560 Nov 03 14:00:04.733750 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.733727 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn"] Nov 03 14:00:04.817290 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.817251 4924 generic.go:356] "Generic (PLEG): container finished" podID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerID="3561cef25b20db900fa4687d96eb64da94d11d0bd25edd6202e5180b4b423877" exitCode=60 Nov 03 14:00:04.817467 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.817343 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" event={"ID":"33af9ebf-6ee4-49be-ae45-1eb279067c83","Type":"ContainerDied","Data":"3561cef25b20db900fa4687d96eb64da94d11d0bd25edd6202e5180b4b423877"} Nov 03 14:00:04.817467 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.817388 4924 scope.go:122] "RemoveContainer" containerID="55cd503ebb77280d371be5665f12aece596ec9410bfce9dd0a84fb9d7ce03449" Nov 03 14:00:04.817816 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.817796 4924 scope.go:122] "RemoveContainer" containerID="3561cef25b20db900fa4687d96eb64da94d11d0bd25edd6202e5180b4b423877" Nov 03 14:00:04.818162 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:04.818141 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p_odh-ai-gateway-infra(33af9ebf-6ee4-49be-ae45-1eb279067c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" Nov 03 14:00:04.823154 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.822414 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" event={"ID":"f0b09327-0a3f-4c6f-be64-211ff018d34f","Type":"ContainerStarted","Data":"5ae4b96c010a0e0378c3fbcc5927f75456814b64e4b4f0ee5036563458fe0762"} Nov 03 14:00:04.823154 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.822463 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" event={"ID":"f0b09327-0a3f-4c6f-be64-211ff018d34f","Type":"ContainerStarted","Data":"6b036747f4b782683033cb74a27b9d566d39a18e82b7bc0dfbd18c1bec009560"} Nov 03 14:00:04.827179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.826534 4924 generic.go:356] "Generic (PLEG): container finished" podID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerID="d0e9c813dfff2e4ab89e2d4538f24dfb4dc89a6a33e52cafbcff95bb8c1443e4" exitCode=60 Nov 03 14:00:04.827179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.826664 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" event={"ID":"538839bf-d1eb-4d7c-a5f1-17e1d28aec78","Type":"ContainerDied","Data":"d0e9c813dfff2e4ab89e2d4538f24dfb4dc89a6a33e52cafbcff95bb8c1443e4"} Nov 03 14:00:04.827179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.827050 4924 scope.go:122] "RemoveContainer" containerID="d0e9c813dfff2e4ab89e2d4538f24dfb4dc89a6a33e52cafbcff95bb8c1443e4" Nov 03 14:00:04.827806 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:04.827778 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568_odh-ai-gateway-infra(538839bf-d1eb-4d7c-a5f1-17e1d28aec78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" Nov 03 14:00:04.830525 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.830488 4924 generic.go:356] "Generic (PLEG): container finished" podID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerID="148219559a906799f599681aed713c1c1e9a3e550f455be57257d0b4c0adbcdb" exitCode=60 Nov 03 14:00:04.830618 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.830582 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" event={"ID":"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe","Type":"ContainerDied","Data":"148219559a906799f599681aed713c1c1e9a3e550f455be57257d0b4c0adbcdb"} Nov 03 14:00:04.830929 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.830913 4924 scope.go:122] "RemoveContainer" containerID="148219559a906799f599681aed713c1c1e9a3e550f455be57257d0b4c0adbcdb" Nov 03 14:00:04.831251 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:04.831229 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq_odh-ai-gateway-infra(8a187191-fbd6-43b3-b9fc-b9c136b9d8fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" Nov 03 14:00:04.832912 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.832890 4924 generic.go:356] "Generic (PLEG): container finished" podID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerID="7dc931c6e129d0a4a02660dffa599d176fee368267ede228d50fa39dd710cf6c" exitCode=60 Nov 03 14:00:04.833023 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.832969 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" event={"ID":"7e3e504b-75b2-4266-88a1-4badcd1407da","Type":"ContainerDied","Data":"7dc931c6e129d0a4a02660dffa599d176fee368267ede228d50fa39dd710cf6c"} Nov 03 14:00:04.833238 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.833200 4924 scope.go:122] "RemoveContainer" containerID="7dc931c6e129d0a4a02660dffa599d176fee368267ede228d50fa39dd710cf6c" Nov 03 14:00:04.833472 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:04.833448 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895240-9vcl7_odh-ai-gateway-infra(7e3e504b-75b2-4266-88a1-4badcd1407da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" Nov 03 14:00:04.835485 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.835460 4924 generic.go:356] "Generic (PLEG): container finished" podID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerID="ab4840641f7f2c252124477fcbb0fe4a0b93d7a3348c34285357ba1c66cc49ef" exitCode=60 Nov 03 14:00:04.835590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.835572 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" event={"ID":"e92ba637-ef5f-4ec7-852f-0966c9357d2d","Type":"ContainerDied","Data":"ab4840641f7f2c252124477fcbb0fe4a0b93d7a3348c34285357ba1c66cc49ef"} Nov 03 14:00:04.835747 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.835728 4924 scope.go:122] "RemoveContainer" containerID="ab4840641f7f2c252124477fcbb0fe4a0b93d7a3348c34285357ba1c66cc49ef" Nov 03 14:00:04.836065 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:04.835983 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh_odh-ai-gateway-infra(e92ba637-ef5f-4ec7-852f-0966c9357d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" Nov 03 14:00:04.836166 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.836128 4924 scope.go:122] "RemoveContainer" containerID="c7a7076cf80c05df33cfac8e4bf94fc376087bd4cc4e3ae220383cef91a9edd3" Nov 03 14:00:04.838308 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.838285 4924 generic.go:356] "Generic (PLEG): container finished" podID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerID="aad5ab9e4763ae5bba019976d01cd0eeca249eb0f6136f123724c06e0a3d51ad" exitCode=60 Nov 03 14:00:04.838392 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.838363 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" event={"ID":"fd8f25c6-1af8-400f-ade0-c41097146ab2","Type":"ContainerDied","Data":"aad5ab9e4763ae5bba019976d01cd0eeca249eb0f6136f123724c06e0a3d51ad"} Nov 03 14:00:04.838555 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.838540 4924 scope.go:122] "RemoveContainer" containerID="aad5ab9e4763ae5bba019976d01cd0eeca249eb0f6136f123724c06e0a3d51ad" Nov 03 14:00:04.839130 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:04.838765 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664_odh-ai-gateway-infra(fd8f25c6-1af8-400f-ade0-c41097146ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" Nov 03 14:00:04.851015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.850994 4924 scope.go:122] "RemoveContainer" containerID="dac77820f50a3ccfaa68f9c353267295950d7f7f10ca70c033e41a27c8f00948" Nov 03 14:00:04.864456 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.864431 4924 scope.go:122] "RemoveContainer" containerID="b4b2899633695361704f8bcbd6c31d0cc79a46d3cc4aa0b3b5d56c61a69606dc" Nov 03 14:00:04.879923 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.879903 4924 scope.go:122] "RemoveContainer" containerID="ffb6049c4efca9dde29bb51845a07be3325b75d15c458cff3a12855b14382e80" Nov 03 14:00:04.893363 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.893335 4924 scope.go:122] "RemoveContainer" containerID="67462e748ac0669a1df38cead91af67f155536bab43d3488d039074fcfb2e8b6" Nov 03 14:00:04.952323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:04.952289 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70a0384-eb61-46e6-8b0d-0105f34de2e3" path="/var/lib/kubelet/pods/d70a0384-eb61-46e6-8b0d-0105f34de2e3/volumes" Nov 03 14:00:05.329205 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:05.329170 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:05.354606 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:05.354553 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" podStartSLOduration=2.354538747 podStartE2EDuration="2.354538747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:05.352522062 +0000 UTC m=+3338.546925662" watchObservedRunningTime="2026-11-03 14:00:05.354538747 +0000 UTC m=+3338.548942341" Nov 03 14:00:05.849175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:05.849140 4924 scope.go:122] "RemoveContainer" containerID="ab4840641f7f2c252124477fcbb0fe4a0b93d7a3348c34285357ba1c66cc49ef" Nov 03 14:00:05.849509 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:05.849483 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh_odh-ai-gateway-infra(e92ba637-ef5f-4ec7-852f-0966c9357d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" Nov 03 14:00:05.852174 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:05.852146 4924 scope.go:122] "RemoveContainer" containerID="aad5ab9e4763ae5bba019976d01cd0eeca249eb0f6136f123724c06e0a3d51ad" Nov 03 14:00:05.852437 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:05.852414 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664_odh-ai-gateway-infra(fd8f25c6-1af8-400f-ade0-c41097146ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" Nov 03 14:00:05.853883 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:05.853863 4924 scope.go:122] "RemoveContainer" containerID="3561cef25b20db900fa4687d96eb64da94d11d0bd25edd6202e5180b4b423877" Nov 03 14:00:05.854169 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:05.854142 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p_odh-ai-gateway-infra(33af9ebf-6ee4-49be-ae45-1eb279067c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" Nov 03 14:00:05.855627 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:05.855610 4924 scope.go:122] "RemoveContainer" containerID="d0e9c813dfff2e4ab89e2d4538f24dfb4dc89a6a33e52cafbcff95bb8c1443e4" Nov 03 14:00:05.855850 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:05.855827 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568_odh-ai-gateway-infra(538839bf-d1eb-4d7c-a5f1-17e1d28aec78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" Nov 03 14:00:05.857216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:05.857199 4924 scope.go:122] "RemoveContainer" containerID="148219559a906799f599681aed713c1c1e9a3e550f455be57257d0b4c0adbcdb" Nov 03 14:00:05.857438 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:05.857412 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq_odh-ai-gateway-infra(8a187191-fbd6-43b3-b9fc-b9c136b9d8fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" Nov 03 14:00:05.858827 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:05.858811 4924 scope.go:122] "RemoveContainer" containerID="7dc931c6e129d0a4a02660dffa599d176fee368267ede228d50fa39dd710cf6c" Nov 03 14:00:05.858987 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:05.858970 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895240-9vcl7_odh-ai-gateway-infra(7e3e504b-75b2-4266-88a1-4badcd1407da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" Nov 03 14:00:06.810893 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:06.810840 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:06.997886 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:06.997849 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:07.141830 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:07.141744 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:08.043542 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:08.043500 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 14:00:08.056404 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:08.056371 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 14:00:08.894424 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:08.894388 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:08.901408 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:08.901379 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 14:00:10.073468 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:10.073434 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:10.254522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:10.254476 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:13.970963 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:13.970921 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:14.329505 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:14.329420 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:14.342097 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:14.342045 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:14.943326 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:14.943293 4924 scope.go:122] "RemoveContainer" containerID="d0e9c813dfff2e4ab89e2d4538f24dfb4dc89a6a33e52cafbcff95bb8c1443e4" Nov 03 14:00:14.943503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:14.943464 4924 scope.go:122] "RemoveContainer" containerID="7dc931c6e129d0a4a02660dffa599d176fee368267ede228d50fa39dd710cf6c" Nov 03 14:00:14.951434 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:14.951412 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:00:15.659936 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:15.659903 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:15.943955 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:15.943837 4924 scope.go:122] "RemoveContainer" containerID="aad5ab9e4763ae5bba019976d01cd0eeca249eb0f6136f123724c06e0a3d51ad" Nov 03 14:00:15.950826 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:15.948793 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" event={"ID":"538839bf-d1eb-4d7c-a5f1-17e1d28aec78","Type":"ContainerStarted","Data":"9e4e996edccf9b2400036a8a23c533445c020a952bb5d48572f98cefad4f6d7e"} Nov 03 14:00:15.950826 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:15.949154 4924 scope.go:122] "RemoveContainer" containerID="9e4e996edccf9b2400036a8a23c533445c020a952bb5d48572f98cefad4f6d7e" Nov 03 14:00:15.950826 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:15.949426 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568_odh-ai-gateway-infra(538839bf-d1eb-4d7c-a5f1-17e1d28aec78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" Nov 03 14:00:15.951427 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:15.951399 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" event={"ID":"7e3e504b-75b2-4266-88a1-4badcd1407da","Type":"ContainerStarted","Data":"19f1c9f9bb44f124ee8928a55fd359abeb98dd8986cbf1fb3431d12851dfeb87"} Nov 03 14:00:16.948144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.948118 4924 scope.go:122] "RemoveContainer" containerID="3561cef25b20db900fa4687d96eb64da94d11d0bd25edd6202e5180b4b423877" Nov 03 14:00:16.948566 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.948208 4924 scope.go:122] "RemoveContainer" containerID="ab4840641f7f2c252124477fcbb0fe4a0b93d7a3348c34285357ba1c66cc49ef" Nov 03 14:00:16.961438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.961411 4924 generic.go:356] "Generic (PLEG): container finished" podID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerID="9e4e996edccf9b2400036a8a23c533445c020a952bb5d48572f98cefad4f6d7e" exitCode=60 Nov 03 14:00:16.961574 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.961482 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" event={"ID":"538839bf-d1eb-4d7c-a5f1-17e1d28aec78","Type":"ContainerDied","Data":"9e4e996edccf9b2400036a8a23c533445c020a952bb5d48572f98cefad4f6d7e"} Nov 03 14:00:16.961574 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.961550 4924 scope.go:122] "RemoveContainer" containerID="d0e9c813dfff2e4ab89e2d4538f24dfb4dc89a6a33e52cafbcff95bb8c1443e4" Nov 03 14:00:16.961903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.961880 4924 scope.go:122] "RemoveContainer" containerID="9e4e996edccf9b2400036a8a23c533445c020a952bb5d48572f98cefad4f6d7e" Nov 03 14:00:16.962181 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:16.962161 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568_odh-ai-gateway-infra(538839bf-d1eb-4d7c-a5f1-17e1d28aec78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" Nov 03 14:00:16.963779 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.963759 4924 generic.go:356] "Generic (PLEG): container finished" podID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerID="19f1c9f9bb44f124ee8928a55fd359abeb98dd8986cbf1fb3431d12851dfeb87" exitCode=60 Nov 03 14:00:16.963871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.963837 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" event={"ID":"7e3e504b-75b2-4266-88a1-4badcd1407da","Type":"ContainerDied","Data":"19f1c9f9bb44f124ee8928a55fd359abeb98dd8986cbf1fb3431d12851dfeb87"} Nov 03 14:00:16.964238 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.964206 4924 scope.go:122] "RemoveContainer" containerID="19f1c9f9bb44f124ee8928a55fd359abeb98dd8986cbf1fb3431d12851dfeb87" Nov 03 14:00:16.964471 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:16.964449 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895240-9vcl7_odh-ai-gateway-infra(7e3e504b-75b2-4266-88a1-4badcd1407da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" Nov 03 14:00:16.968489 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.968355 4924 generic.go:356] "Generic (PLEG): container finished" podID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerID="38ec3f03b09cb75a7ced86918eced8da5a65d24705ac7c2879fc134f8637a3d7" exitCode=60 Nov 03 14:00:16.968555 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.968497 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" event={"ID":"fd8f25c6-1af8-400f-ade0-c41097146ab2","Type":"ContainerDied","Data":"38ec3f03b09cb75a7ced86918eced8da5a65d24705ac7c2879fc134f8637a3d7"} Nov 03 14:00:16.970932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.970913 4924 scope.go:122] "RemoveContainer" containerID="38ec3f03b09cb75a7ced86918eced8da5a65d24705ac7c2879fc134f8637a3d7" Nov 03 14:00:16.971173 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:16.971149 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664_odh-ai-gateway-infra(fd8f25c6-1af8-400f-ade0-c41097146ab2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" Nov 03 14:00:16.992914 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:16.992889 4924 scope.go:122] "RemoveContainer" containerID="7dc931c6e129d0a4a02660dffa599d176fee368267ede228d50fa39dd710cf6c" Nov 03 14:00:17.008837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.008819 4924 scope.go:122] "RemoveContainer" containerID="aad5ab9e4763ae5bba019976d01cd0eeca249eb0f6136f123724c06e0a3d51ad" Nov 03 14:00:17.084576 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.084546 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568"] Nov 03 14:00:17.199126 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.199003 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7"] Nov 03 14:00:17.978951 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.978908 4924 generic.go:356] "Generic (PLEG): container finished" podID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerID="7750919837c7771f38d0fafdbc7116633ee0ef08ab5002207667c9a3294fd4f5" exitCode=60 Nov 03 14:00:17.979335 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.978971 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" event={"ID":"e92ba637-ef5f-4ec7-852f-0966c9357d2d","Type":"ContainerDied","Data":"7750919837c7771f38d0fafdbc7116633ee0ef08ab5002207667c9a3294fd4f5"} Nov 03 14:00:17.979335 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.979044 4924 scope.go:122] "RemoveContainer" containerID="ab4840641f7f2c252124477fcbb0fe4a0b93d7a3348c34285357ba1c66cc49ef" Nov 03 14:00:17.979583 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.979563 4924 scope.go:122] "RemoveContainer" containerID="7750919837c7771f38d0fafdbc7116633ee0ef08ab5002207667c9a3294fd4f5" Nov 03 14:00:17.979919 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:17.979883 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh_odh-ai-gateway-infra(e92ba637-ef5f-4ec7-852f-0966c9357d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" Nov 03 14:00:17.983208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.983180 4924 generic.go:356] "Generic (PLEG): container finished" podID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerID="e088a0ec226baad275c834fcb131dc4669a7fc20a329db3b2016cc90f838b10f" exitCode=60 Nov 03 14:00:17.983298 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.983262 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" event={"ID":"33af9ebf-6ee4-49be-ae45-1eb279067c83","Type":"ContainerDied","Data":"e088a0ec226baad275c834fcb131dc4669a7fc20a329db3b2016cc90f838b10f"} Nov 03 14:00:17.983686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:17.983666 4924 scope.go:122] "RemoveContainer" containerID="e088a0ec226baad275c834fcb131dc4669a7fc20a329db3b2016cc90f838b10f" Nov 03 14:00:17.984007 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:17.983984 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p_odh-ai-gateway-infra(33af9ebf-6ee4-49be-ae45-1eb279067c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" Nov 03 14:00:18.003732 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.003704 4924 scope.go:122] "RemoveContainer" containerID="3561cef25b20db900fa4687d96eb64da94d11d0bd25edd6202e5180b4b423877" Nov 03 14:00:18.165838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.165805 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:18.202125 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.202091 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664"] Nov 03 14:00:18.211973 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.211948 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:18.294982 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.294948 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3e504b-75b2-4266-88a1-4badcd1407da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e504b-75b2-4266-88a1-4badcd1407da-maas-api-service-ca\") pod \"7e3e504b-75b2-4266-88a1-4badcd1407da\" (UID: \"7e3e504b-75b2-4266-88a1-4badcd1407da\") " Nov 03 14:00:18.295208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.295098 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3e504b-75b2-4266-88a1-4badcd1407da-kube-api-access-8l8v4\" (UniqueName: \"kubernetes.io/projected/7e3e504b-75b2-4266-88a1-4badcd1407da-kube-api-access-8l8v4\") pod \"7e3e504b-75b2-4266-88a1-4badcd1407da\" (UID: \"7e3e504b-75b2-4266-88a1-4badcd1407da\") " Nov 03 14:00:18.295438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.295411 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3e504b-75b2-4266-88a1-4badcd1407da-maas-api-service-ca" pod "7e3e504b-75b2-4266-88a1-4badcd1407da" (UID: "7e3e504b-75b2-4266-88a1-4badcd1407da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:18.295574 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.295555 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e504b-75b2-4266-88a1-4badcd1407da-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.297398 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.297161 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3e504b-75b2-4266-88a1-4badcd1407da-kube-api-access-8l8v4" pod "7e3e504b-75b2-4266-88a1-4badcd1407da" (UID: "7e3e504b-75b2-4266-88a1-4badcd1407da"). InnerVolumeSpecName "kube-api-access-8l8v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:18.337889 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.337859 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:18.396432 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.396397 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-kube-api-access-b57hr\" (UniqueName: \"kubernetes.io/projected/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-kube-api-access-b57hr\") pod \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\" (UID: \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\") " Nov 03 14:00:18.396600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.396446 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-maas-api-service-ca\") pod \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\" (UID: \"538839bf-d1eb-4d7c-a5f1-17e1d28aec78\") " Nov 03 14:00:18.396958 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.396928 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-maas-api-service-ca" pod "538839bf-d1eb-4d7c-a5f1-17e1d28aec78" (UID: "538839bf-d1eb-4d7c-a5f1-17e1d28aec78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:18.397060 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.396958 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8v4\" (UniqueName: \"kubernetes.io/projected/7e3e504b-75b2-4266-88a1-4badcd1407da-kube-api-access-8l8v4\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.398658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.398634 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-kube-api-access-b57hr" pod "538839bf-d1eb-4d7c-a5f1-17e1d28aec78" (UID: "538839bf-d1eb-4d7c-a5f1-17e1d28aec78"). InnerVolumeSpecName "kube-api-access-b57hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:18.497976 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.497937 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8f25c6-1af8-400f-ade0-c41097146ab2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f25c6-1af8-400f-ade0-c41097146ab2-maas-api-service-ca\") pod \"fd8f25c6-1af8-400f-ade0-c41097146ab2\" (UID: \"fd8f25c6-1af8-400f-ade0-c41097146ab2\") " Nov 03 14:00:18.498218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.498200 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f25c6-1af8-400f-ade0-c41097146ab2-kube-api-access-fq4bz\" (UniqueName: \"kubernetes.io/projected/fd8f25c6-1af8-400f-ade0-c41097146ab2-kube-api-access-fq4bz\") pod \"fd8f25c6-1af8-400f-ade0-c41097146ab2\" (UID: \"fd8f25c6-1af8-400f-ade0-c41097146ab2\") " Nov 03 14:00:18.498398 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.498369 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8f25c6-1af8-400f-ade0-c41097146ab2-maas-api-service-ca" pod "fd8f25c6-1af8-400f-ade0-c41097146ab2" (UID: "fd8f25c6-1af8-400f-ade0-c41097146ab2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:18.498592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.498575 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b57hr\" (UniqueName: \"kubernetes.io/projected/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-kube-api-access-b57hr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.498669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.498597 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538839bf-d1eb-4d7c-a5f1-17e1d28aec78-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.498669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.498612 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd8f25c6-1af8-400f-ade0-c41097146ab2-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.500267 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.500247 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f25c6-1af8-400f-ade0-c41097146ab2-kube-api-access-fq4bz" pod "fd8f25c6-1af8-400f-ade0-c41097146ab2" (UID: "fd8f25c6-1af8-400f-ade0-c41097146ab2"). InnerVolumeSpecName "kube-api-access-fq4bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:18.567035 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:18.566944 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:00:18.600097 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.599890 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4bz\" (UniqueName: \"kubernetes.io/projected/fd8f25c6-1af8-400f-ade0-c41097146ab2-kube-api-access-fq4bz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.943668 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.943633 4924 scope.go:122] "RemoveContainer" containerID="148219559a906799f599681aed713c1c1e9a3e550f455be57257d0b4c0adbcdb" Nov 03 14:00:18.993701 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.993676 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" Nov 03 14:00:18.993701 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.993685 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568" event={"ID":"538839bf-d1eb-4d7c-a5f1-17e1d28aec78","Type":"ContainerDied","Data":"7b65979a91213cf19d7b1a4ea3c4720fbc1d03e04ea2a4ce6577a8df4fa8b07e"} Nov 03 14:00:18.994103 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.993744 4924 scope.go:122] "RemoveContainer" containerID="9e4e996edccf9b2400036a8a23c533445c020a952bb5d48572f98cefad4f6d7e" Nov 03 14:00:18.995045 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.995028 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" Nov 03 14:00:18.995147 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.995052 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7" event={"ID":"7e3e504b-75b2-4266-88a1-4badcd1407da","Type":"ContainerDied","Data":"d06198b57f8a81fa1b42ae8b9c30ab0af7fb360dd06b4dc2e5dc8618fcbc27bb"} Nov 03 14:00:18.999505 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.998789 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" event={"ID":"fd8f25c6-1af8-400f-ade0-c41097146ab2","Type":"ContainerDied","Data":"4a52317fb3ec6e16dbbd82dda78acc07ac62120ba3cf8bbcf7fd805ead0815b4"} Nov 03 14:00:18.999505 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:18.998900 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664" Nov 03 14:00:19.005520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.005501 4924 scope.go:122] "RemoveContainer" containerID="19f1c9f9bb44f124ee8928a55fd359abeb98dd8986cbf1fb3431d12851dfeb87" Nov 03 14:00:19.017994 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.017966 4924 scope.go:122] "RemoveContainer" containerID="38ec3f03b09cb75a7ced86918eced8da5a65d24705ac7c2879fc134f8637a3d7" Nov 03 14:00:19.153390 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.152413 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664"] Nov 03 14:00:19.157548 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.157520 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895240-k7664"] Nov 03 14:00:19.173406 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.173371 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:19.269930 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.269894 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568"] Nov 03 14:00:19.286854 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.286821 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895240-th568"] Nov 03 14:00:19.335759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.335726 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh"] Nov 03 14:00:19.401300 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.398548 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7"] Nov 03 14:00:19.412279 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.412244 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-9vcl7"] Nov 03 14:00:19.423924 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.423879 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p"] Nov 03 14:00:19.515039 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.514976 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:19.596713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.596684 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:19.612109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.612060 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92ba637-ef5f-4ec7-852f-0966c9357d2d-kube-api-access-79wnm\" (UniqueName: \"kubernetes.io/projected/e92ba637-ef5f-4ec7-852f-0966c9357d2d-kube-api-access-79wnm\") pod \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\" (UID: \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\") " Nov 03 14:00:19.612387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.612363 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92ba637-ef5f-4ec7-852f-0966c9357d2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba637-ef5f-4ec7-852f-0966c9357d2d-maas-api-service-ca\") pod \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\" (UID: \"e92ba637-ef5f-4ec7-852f-0966c9357d2d\") " Nov 03 14:00:19.612710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.612690 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ba637-ef5f-4ec7-852f-0966c9357d2d-maas-api-service-ca" pod "e92ba637-ef5f-4ec7-852f-0966c9357d2d" (UID: "e92ba637-ef5f-4ec7-852f-0966c9357d2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:19.614028 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.614010 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92ba637-ef5f-4ec7-852f-0966c9357d2d-kube-api-access-79wnm" pod "e92ba637-ef5f-4ec7-852f-0966c9357d2d" (UID: "e92ba637-ef5f-4ec7-852f-0966c9357d2d"). InnerVolumeSpecName "kube-api-access-79wnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:19.713815 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.713782 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33af9ebf-6ee4-49be-ae45-1eb279067c83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33af9ebf-6ee4-49be-ae45-1eb279067c83-maas-api-service-ca\") pod \"33af9ebf-6ee4-49be-ae45-1eb279067c83\" (UID: \"33af9ebf-6ee4-49be-ae45-1eb279067c83\") " Nov 03 14:00:19.713815 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.713826 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33af9ebf-6ee4-49be-ae45-1eb279067c83-kube-api-access-5kksc\" (UniqueName: \"kubernetes.io/projected/33af9ebf-6ee4-49be-ae45-1eb279067c83-kube-api-access-5kksc\") pod \"33af9ebf-6ee4-49be-ae45-1eb279067c83\" (UID: \"33af9ebf-6ee4-49be-ae45-1eb279067c83\") " Nov 03 14:00:19.714321 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.714266 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33af9ebf-6ee4-49be-ae45-1eb279067c83-maas-api-service-ca" pod "33af9ebf-6ee4-49be-ae45-1eb279067c83" (UID: "33af9ebf-6ee4-49be-ae45-1eb279067c83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:19.714422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.714355 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92ba637-ef5f-4ec7-852f-0966c9357d2d-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:19.714422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.714370 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wnm\" (UniqueName: \"kubernetes.io/projected/e92ba637-ef5f-4ec7-852f-0966c9357d2d-kube-api-access-79wnm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:19.716058 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.716035 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33af9ebf-6ee4-49be-ae45-1eb279067c83-kube-api-access-5kksc" pod "33af9ebf-6ee4-49be-ae45-1eb279067c83" (UID: "33af9ebf-6ee4-49be-ae45-1eb279067c83"). InnerVolumeSpecName "kube-api-access-5kksc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:19.816059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.815984 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kksc\" (UniqueName: \"kubernetes.io/projected/33af9ebf-6ee4-49be-ae45-1eb279067c83-kube-api-access-5kksc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:19.816059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:19.816022 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33af9ebf-6ee4-49be-ae45-1eb279067c83-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:20.011181 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.011147 4924 generic.go:356] "Generic (PLEG): container finished" podID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerID="80f85244417f2f1bbb311636f6cb0c7e69493f2ce4d5dd9cd3340cf9dd722bd2" exitCode=60 Nov 03 14:00:20.011650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.011219 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" event={"ID":"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe","Type":"ContainerDied","Data":"80f85244417f2f1bbb311636f6cb0c7e69493f2ce4d5dd9cd3340cf9dd722bd2"} Nov 03 14:00:20.011650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.011275 4924 scope.go:122] "RemoveContainer" containerID="148219559a906799f599681aed713c1c1e9a3e550f455be57257d0b4c0adbcdb" Nov 03 14:00:20.011792 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.011691 4924 scope.go:122] "RemoveContainer" containerID="80f85244417f2f1bbb311636f6cb0c7e69493f2ce4d5dd9cd3340cf9dd722bd2" Nov 03 14:00:20.012030 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:20.012002 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq_odh-ai-gateway-infra(8a187191-fbd6-43b3-b9fc-b9c136b9d8fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" Nov 03 14:00:20.014176 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.014156 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" Nov 03 14:00:20.014176 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.014166 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh" event={"ID":"e92ba637-ef5f-4ec7-852f-0966c9357d2d","Type":"ContainerDied","Data":"910f1299f43413ab1c956b81c54000a3367c7a18286655f294cc02444bbd9d1f"} Nov 03 14:00:20.016744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.016720 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" event={"ID":"33af9ebf-6ee4-49be-ae45-1eb279067c83","Type":"ContainerDied","Data":"d8fe83a10839f9cf86eafdf73791cd68b4f82fd1a378cda3dfb88364190a6f92"} Nov 03 14:00:20.016846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.016732 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p" Nov 03 14:00:20.025950 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.025934 4924 scope.go:122] "RemoveContainer" containerID="7750919837c7771f38d0fafdbc7116633ee0ef08ab5002207667c9a3294fd4f5" Nov 03 14:00:20.043533 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.042942 4924 scope.go:122] "RemoveContainer" containerID="e088a0ec226baad275c834fcb131dc4669a7fc20a329db3b2016cc90f838b10f" Nov 03 14:00:20.226062 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.226033 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh"] Nov 03 14:00:20.230051 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.230015 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895240-2wjbh"] Nov 03 14:00:20.332443 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.332405 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p"] Nov 03 14:00:20.338560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.338528 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895240-x9m5p"] Nov 03 14:00:20.968413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.968381 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" path="/var/lib/kubelet/pods/33af9ebf-6ee4-49be-ae45-1eb279067c83/volumes" Nov 03 14:00:20.968817 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.968803 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" path="/var/lib/kubelet/pods/538839bf-d1eb-4d7c-a5f1-17e1d28aec78/volumes" Nov 03 14:00:20.969204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.969189 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" path="/var/lib/kubelet/pods/7e3e504b-75b2-4266-88a1-4badcd1407da/volumes" Nov 03 14:00:20.969887 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.969871 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" path="/var/lib/kubelet/pods/e92ba637-ef5f-4ec7-852f-0966c9357d2d/volumes" Nov 03 14:00:20.970276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:20.970260 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" path="/var/lib/kubelet/pods/fd8f25c6-1af8-400f-ade0-c41097146ab2/volumes" Nov 03 14:00:21.132247 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:21.132219 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq"] Nov 03 14:00:21.305313 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:21.305288 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:21.332819 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:21.332780 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-kube-api-access-2b5rp\" (UniqueName: \"kubernetes.io/projected/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-kube-api-access-2b5rp\") pod \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\" (UID: \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\") " Nov 03 14:00:21.334984 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:21.334953 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-kube-api-access-2b5rp" pod "8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" (UID: "8a187191-fbd6-43b3-b9fc-b9c136b9d8fe"). InnerVolumeSpecName "kube-api-access-2b5rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:21.434148 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:21.434108 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-maas-api-service-ca\") pod \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\" (UID: \"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe\") " Nov 03 14:00:21.434656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:21.434634 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b5rp\" (UniqueName: \"kubernetes.io/projected/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-kube-api-access-2b5rp\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:21.435089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:21.435043 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-maas-api-service-ca" pod "8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" (UID: "8a187191-fbd6-43b3-b9fc-b9c136b9d8fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:21.536295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:21.536197 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:22.037690 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:22.037653 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" Nov 03 14:00:22.037866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:22.037682 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq" event={"ID":"8a187191-fbd6-43b3-b9fc-b9c136b9d8fe","Type":"ContainerDied","Data":"28a32d3a800aae281619feea8e4810d274802dc6f5c697d44ac8bb9b774c2f04"} Nov 03 14:00:22.037866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:22.037759 4924 scope.go:122] "RemoveContainer" containerID="80f85244417f2f1bbb311636f6cb0c7e69493f2ce4d5dd9cd3340cf9dd722bd2" Nov 03 14:00:22.194596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:22.194561 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq"] Nov 03 14:00:22.242599 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:22.242560 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895240-lwvlq"] Nov 03 14:00:22.951984 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:22.951954 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" path="/var/lib/kubelet/pods/8a187191-fbd6-43b3-b9fc-b9c136b9d8fe/volumes" Nov 03 14:00:23.952999 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:23.952850 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:24.323446 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:24.323354 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:29.395132 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:29.395063 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:34.451678 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:34.451645 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:35.457692 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:35.457654 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:39.492865 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:39.492788 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:40.581867 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:40.581831 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:40.923567 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:40.923533 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:41.067839 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:41.067802 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:44.690741 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:44.690709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:46.053777 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:46.053734 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn"] Nov 03 14:00:46.148052 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:46.148005 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:48.558509 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:00:48.558457 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:00:49.877166 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:49.877118 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:52.617599 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:52.617563 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:00:55.002733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:00:55.002697 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:00.049942 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:00.049890 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:00.198652 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:00.198618 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:00.331676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:00.331578 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:01.078911 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.078865 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" podUID="f0b09327-0a3f-4c6f-be64-211ff018d34f" containerName="main" containerID="cri-o://5ae4b96c010a0e0378c3fbcc5927f75456814b64e4b4f0ee5036563458fe0762" gracePeriod=45 Nov 03 14:01:01.316733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.316690 4924 generic.go:356] "Generic (PLEG): container finished" podID="f0b09327-0a3f-4c6f-be64-211ff018d34f" containerID="5ae4b96c010a0e0378c3fbcc5927f75456814b64e4b4f0ee5036563458fe0762" exitCode=0 Nov 03 14:01:01.316906 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.316760 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" event={"ID":"f0b09327-0a3f-4c6f-be64-211ff018d34f","Type":"ContainerDied","Data":"5ae4b96c010a0e0378c3fbcc5927f75456814b64e4b4f0ee5036563458fe0762"} Nov 03 14:01:01.355720 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.355686 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:01:01.396306 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396273 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b09327-0a3f-4c6f-be64-211ff018d34f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b09327-0a3f-4c6f-be64-211ff018d34f-tls-certs\") pod \"f0b09327-0a3f-4c6f-be64-211ff018d34f\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " Nov 03 14:01:01.396459 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396327 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-model-cache\") pod \"f0b09327-0a3f-4c6f-be64-211ff018d34f\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " Nov 03 14:01:01.396459 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396406 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-tmp-dir\") pod \"f0b09327-0a3f-4c6f-be64-211ff018d34f\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " Nov 03 14:01:01.396459 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396439 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-dshm\") pod \"f0b09327-0a3f-4c6f-be64-211ff018d34f\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " Nov 03 14:01:01.396577 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396503 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-home\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-home\") pod \"f0b09327-0a3f-4c6f-be64-211ff018d34f\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " Nov 03 14:01:01.396638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396589 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-tmp-dir" pod "f0b09327-0a3f-4c6f-be64-211ff018d34f" (UID: "f0b09327-0a3f-4c6f-be64-211ff018d34f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:01:01.396638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396604 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-model-cache" pod "f0b09327-0a3f-4c6f-be64-211ff018d34f" (UID: "f0b09327-0a3f-4c6f-be64-211ff018d34f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:01:01.396638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396622 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b09327-0a3f-4c6f-be64-211ff018d34f-kube-api-access-qpw2d\" (UniqueName: \"kubernetes.io/projected/f0b09327-0a3f-4c6f-be64-211ff018d34f-kube-api-access-qpw2d\") pod \"f0b09327-0a3f-4c6f-be64-211ff018d34f\" (UID: \"f0b09327-0a3f-4c6f-be64-211ff018d34f\") " Nov 03 14:01:01.396772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.396749 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-home" pod "f0b09327-0a3f-4c6f-be64-211ff018d34f" (UID: "f0b09327-0a3f-4c6f-be64-211ff018d34f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:01:01.397030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.397007 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:01.397030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.397029 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:01.397249 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.397037 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:01.398629 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.398601 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-dshm" pod "f0b09327-0a3f-4c6f-be64-211ff018d34f" (UID: "f0b09327-0a3f-4c6f-be64-211ff018d34f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:01:01.398714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.398655 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b09327-0a3f-4c6f-be64-211ff018d34f-kube-api-access-qpw2d" pod "f0b09327-0a3f-4c6f-be64-211ff018d34f" (UID: "f0b09327-0a3f-4c6f-be64-211ff018d34f"). InnerVolumeSpecName "kube-api-access-qpw2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:01:01.398714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.398695 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b09327-0a3f-4c6f-be64-211ff018d34f-tls-certs" pod "f0b09327-0a3f-4c6f-be64-211ff018d34f" (UID: "f0b09327-0a3f-4c6f-be64-211ff018d34f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:01:01.498107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.498060 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b09327-0a3f-4c6f-be64-211ff018d34f-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:01.498107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.498104 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpw2d\" (UniqueName: \"kubernetes.io/projected/f0b09327-0a3f-4c6f-be64-211ff018d34f-kube-api-access-qpw2d\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:01.498107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:01.498115 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b09327-0a3f-4c6f-be64-211ff018d34f-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:02.324987 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:02.324954 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" Nov 03 14:01:02.325453 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:02.324982 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn" event={"ID":"f0b09327-0a3f-4c6f-be64-211ff018d34f","Type":"ContainerDied","Data":"6b036747f4b782683033cb74a27b9d566d39a18e82b7bc0dfbd18c1bec009560"} Nov 03 14:01:02.325453 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:02.325033 4924 scope.go:122] "RemoveContainer" containerID="5ae4b96c010a0e0378c3fbcc5927f75456814b64e4b4f0ee5036563458fe0762" Nov 03 14:01:02.480940 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:02.480903 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn"] Nov 03 14:01:02.527241 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:02.527201 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-921466/noten-model-921466-kserve-64d75f5cfc-vwwtn"] Nov 03 14:01:02.953682 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:02.953641 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b09327-0a3f-4c6f-be64-211ff018d34f" path="/var/lib/kubelet/pods/f0b09327-0a3f-4c6f-be64-211ff018d34f/volumes" Nov 03 14:01:05.349862 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:05.349821 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:10.476833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:10.476793 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:15.577474 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:15.577435 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:18.558325 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:01:18.558285 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:01:20.675760 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:20.675703 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:23.394358 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:23.394317 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:25.166561 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.166529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:25.437111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437011 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s"] Nov 03 14:01:25.437515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437491 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:25.437515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437506 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437520 4924 cpu_manager.go:414] "containerMap: removing container" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437525 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437534 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437540 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437553 4924 cpu_manager.go:414] "containerMap: removing container" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437559 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437567 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f0b09327-0a3f-4c6f-be64-211ff018d34f" containerName="main" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437572 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b09327-0a3f-4c6f-be64-211ff018d34f" containerName="main" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437581 4924 cpu_manager.go:414] "containerMap: removing container" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437586 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437598 4924 cpu_manager.go:414] "containerMap: removing container" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437602 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437610 4924 cpu_manager.go:414] "containerMap: removing container" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437615 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437622 4924 cpu_manager.go:414] "containerMap: removing container" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437627 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437636 4924 cpu_manager.go:414] "containerMap: removing container" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437640 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437648 4924 cpu_manager.go:414] "containerMap: removing container" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437653 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437658 4924 cpu_manager.go:414] "containerMap: removing container" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.437651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437663 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437677 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437682 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437689 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437693 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437700 4924 cpu_manager.go:414] "containerMap: removing container" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437704 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437799 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437809 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437821 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437831 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b09327-0a3f-4c6f-be64-211ff018d34f" containerName="main" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437837 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437843 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437848 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437853 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437858 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437864 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437870 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8f25c6-1af8-400f-ade0-c41097146ab2" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437877 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437881 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:25.438864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.437886 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:25.441517 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.441496 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.462430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.462398 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.462578 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.462559 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.462646 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.462588 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9vd\" (UniqueName: \"kubernetes.io/projected/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-kube-api-access-7m9vd\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.462773 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.462754 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-metrics-certs\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.494459 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.494430 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c4e398-serving-cert\"" Nov 03 14:01:25.496155 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.496122 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s"] Nov 03 14:01:25.563863 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.563822 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-metrics-certs\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.563863 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.563866 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.564193 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.563933 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.564193 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.563950 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9vd\" (UniqueName: \"kubernetes.io/projected/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-kube-api-access-7m9vd\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.564354 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:01:25.564040 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-serving-cert: secret "maas-api-e2e-extmod-a-c4e398-serving-cert" not found Nov 03 14:01:25.564475 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:01:25.564456 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls podName:f627e6ce-9bc2-4c69-abb4-8fa3df2a9862 nodeName:}" failed. No retries permitted until 2026-11-03 14:01:26.064433231 +0000 UTC m=+3419.258836818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls") pod "maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" (UID: "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862") : secret "maas-api-e2e-extmod-a-c4e398-serving-cert" not found Nov 03 14:01:25.564952 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.564924 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.566607 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.566584 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-metrics-certs\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.619975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.619942 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9vd\" (UniqueName: \"kubernetes.io/projected/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-kube-api-access-7m9vd\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:25.633518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.633484 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:25.821803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:25.821723 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:26.071328 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:26.071286 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:26.074028 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:26.073963 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:26.097090 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:26.096179 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:26.357116 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:26.357005 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:26.750518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:26.749108 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s"] Nov 03 14:01:26.752806 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:01:26.752770 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf627e6ce_9bc2_4c69_abb4_8fa3df2a9862.slice/crio-96b1fd3c8ac813ecb00885b633661250308e049a1d24c521845eb3fab99e7a46 WatchSource:0}: Error finding container 96b1fd3c8ac813ecb00885b633661250308e049a1d24c521845eb3fab99e7a46: Status 404 returned error can't find the container with id 96b1fd3c8ac813ecb00885b633661250308e049a1d24c521845eb3fab99e7a46 Nov 03 14:01:26.754367 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:26.754345 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:01:26.854058 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:26.854022 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:27.514925 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:27.514881 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" event={"ID":"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862","Type":"ContainerStarted","Data":"96b1fd3c8ac813ecb00885b633661250308e049a1d24c521845eb3fab99e7a46"} Nov 03 14:01:28.523295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:28.523256 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" event={"ID":"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862","Type":"ContainerStarted","Data":"63747fb8e4659b16cd78075f634a49cf418dd6935c7f6fa9d6d26e506acd2c90"} Nov 03 14:01:28.523806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:28.523339 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:28.645408 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:28.645340 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" podStartSLOduration=2.6942897009999998 podStartE2EDuration="3.64531818s" totalImagesPullingTime="951.028479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:01:28.634806413 +0000 UTC m=+3421.829210032" watchObservedRunningTime="2026-11-03 14:01:28.64531818 +0000 UTC m=+3421.839721774" Nov 03 14:01:30.203846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:30.203806 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:30.390063 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:30.390026 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:30.532619 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:30.532525 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:31.337452 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:31.337417 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:31.672337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:31.672304 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:31.839652 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:31.839617 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:34.466450 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:34.466409 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:34.537007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:34.536979 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:01:34.639015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:34.638975 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:35.784693 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:35.784659 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:36.902271 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:36.902235 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:37.024940 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:37.024905 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:42.624283 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:42.624237 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:47.157245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.157208 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf"] Nov 03 14:01:47.157919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.157896 4924 cpu_manager.go:414] "containerMap: removing container" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:47.158030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.157920 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:47.158030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.157981 4924 cpu_manager.go:414] "containerMap: removing container" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:47.158030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.157992 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:47.158030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158010 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:47.158030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158018 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:47.158319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158035 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:47.158319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158043 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:47.158319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158203 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3e504b-75b2-4266-88a1-4badcd1407da" containerName="cleanup" Nov 03 14:01:47.158319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158220 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="33af9ebf-6ee4-49be-ae45-1eb279067c83" containerName="cleanup" Nov 03 14:01:47.158319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158233 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92ba637-ef5f-4ec7-852f-0966c9357d2d" containerName="cleanup" Nov 03 14:01:47.158319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158244 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="538839bf-d1eb-4d7c-a5f1-17e1d28aec78" containerName="cleanup" Nov 03 14:01:47.158319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.158256 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a187191-fbd6-43b3-b9fc-b9c136b9d8fe" containerName="cleanup" Nov 03 14:01:47.161878 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.161857 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.248319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.248279 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.248528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.248493 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.248667 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.248638 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmpw\" (UniqueName: \"kubernetes.io/projected/8abd6683-d58f-4211-ad1f-cbc91b261196-kube-api-access-8hmpw\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.258122 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.258091 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:01:47.258346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.258121 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:01:47.261816 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.261789 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf"] Nov 03 14:01:47.349772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.349728 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.349990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.349837 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmpw\" (UniqueName: \"kubernetes.io/projected/8abd6683-d58f-4211-ad1f-cbc91b261196-kube-api-access-8hmpw\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.349990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.349915 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.350638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.350610 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.352119 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.352097 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.404127 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.404091 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmpw\" (UniqueName: \"kubernetes.io/projected/8abd6683-d58f-4211-ad1f-cbc91b261196-kube-api-access-8hmpw\") pod \"maas-api-revoke-sub-1f992c0e-ch4qf\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.475988 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.475905 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:47.845169 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:01:47.845132 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abd6683_d58f_4211_ad1f_cbc91b261196.slice/crio-81c6695f7ba6c05c50ce675c8485f0f38b9a118e9f987515ec5487f586f8c550 WatchSource:0}: Error finding container 81c6695f7ba6c05c50ce675c8485f0f38b9a118e9f987515ec5487f586f8c550: Status 404 returned error can't find the container with id 81c6695f7ba6c05c50ce675c8485f0f38b9a118e9f987515ec5487f586f8c550 Nov 03 14:01:47.845776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.845750 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf"] Nov 03 14:01:47.868969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:47.868944 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:48.560690 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:01:48.560608 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:01:48.666243 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:48.666197 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" event={"ID":"8abd6683-d58f-4211-ad1f-cbc91b261196","Type":"ContainerStarted","Data":"c775f9b6a26e3320190f9bdcceb682bb6b6b912fe0f14a64f16f2a92f0f6de53"} Nov 03 14:01:48.666446 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:48.666256 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" event={"ID":"8abd6683-d58f-4211-ad1f-cbc91b261196","Type":"ContainerStarted","Data":"81c6695f7ba6c05c50ce675c8485f0f38b9a118e9f987515ec5487f586f8c550"} Nov 03 14:01:48.776571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:48.776508 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" podStartSLOduration=1.7764936630000001 podStartE2EDuration="1.776493663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:01:48.772661256 +0000 UTC m=+3441.967064851" watchObservedRunningTime="2026-11-03 14:01:48.776493663 +0000 UTC m=+3441.970897256" Nov 03 14:01:49.673988 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:49.673953 4924 generic.go:356] "Generic (PLEG): container finished" podID="8abd6683-d58f-4211-ad1f-cbc91b261196" containerID="c775f9b6a26e3320190f9bdcceb682bb6b6b912fe0f14a64f16f2a92f0f6de53" exitCode=0 Nov 03 14:01:49.674435 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:49.674005 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" event={"ID":"8abd6683-d58f-4211-ad1f-cbc91b261196","Type":"ContainerDied","Data":"c775f9b6a26e3320190f9bdcceb682bb6b6b912fe0f14a64f16f2a92f0f6de53"} Nov 03 14:01:50.831366 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.831340 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:50.889909 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.889876 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-openshift-ca\") pod \"8abd6683-d58f-4211-ad1f-cbc91b261196\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " Nov 03 14:01:50.890106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.890009 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abd6683-d58f-4211-ad1f-cbc91b261196-kube-api-access-8hmpw\" (UniqueName: \"kubernetes.io/projected/8abd6683-d58f-4211-ad1f-cbc91b261196-kube-api-access-8hmpw\") pod \"8abd6683-d58f-4211-ad1f-cbc91b261196\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " Nov 03 14:01:50.890106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.890052 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-xks-ca\") pod \"8abd6683-d58f-4211-ad1f-cbc91b261196\" (UID: \"8abd6683-d58f-4211-ad1f-cbc91b261196\") " Nov 03 14:01:50.890306 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.890283 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-openshift-ca" pod "8abd6683-d58f-4211-ad1f-cbc91b261196" (UID: "8abd6683-d58f-4211-ad1f-cbc91b261196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:01:50.890481 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.890465 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:50.892166 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.892137 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-xks-ca" pod "8abd6683-d58f-4211-ad1f-cbc91b261196" (UID: "8abd6683-d58f-4211-ad1f-cbc91b261196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:01:50.892295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.892273 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abd6683-d58f-4211-ad1f-cbc91b261196-kube-api-access-8hmpw" pod "8abd6683-d58f-4211-ad1f-cbc91b261196" (UID: "8abd6683-d58f-4211-ad1f-cbc91b261196"). InnerVolumeSpecName "kube-api-access-8hmpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:01:50.991871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.991838 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hmpw\" (UniqueName: \"kubernetes.io/projected/8abd6683-d58f-4211-ad1f-cbc91b261196-kube-api-access-8hmpw\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:50.991871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:50.991870 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd6683-d58f-4211-ad1f-cbc91b261196-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:51.690584 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:51.690551 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" Nov 03 14:01:51.690772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:51.690552 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf" event={"ID":"8abd6683-d58f-4211-ad1f-cbc91b261196","Type":"ContainerDied","Data":"81c6695f7ba6c05c50ce675c8485f0f38b9a118e9f987515ec5487f586f8c550"} Nov 03 14:01:51.690772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:51.690685 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81c6695f7ba6c05c50ce675c8485f0f38b9a118e9f987515ec5487f586f8c550" Nov 03 14:01:53.008886 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:53.008849 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:58.011484 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.011449 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8"] Nov 03 14:01:58.012153 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.012131 4924 cpu_manager.go:414] "containerMap: removing container" podUID="8abd6683-d58f-4211-ad1f-cbc91b261196" containerName="revoke-keys" Nov 03 14:01:58.012153 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.012153 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abd6683-d58f-4211-ad1f-cbc91b261196" containerName="revoke-keys" Nov 03 14:01:58.012351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.012336 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abd6683-d58f-4211-ad1f-cbc91b261196" containerName="revoke-keys" Nov 03 14:01:58.016676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.016657 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.066081 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.066018 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgrr\" (UniqueName: \"kubernetes.io/projected/93fdfdc7-e018-42f1-8a2d-03f87df6f493-kube-api-access-2wgrr\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.066370 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.066341 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.066516 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.066463 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.066795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.066773 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:01:58.113408 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.113364 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:01:58.117276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.117248 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8"] Nov 03 14:01:58.166559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.166524 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:01:58.167156 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.167131 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgrr\" (UniqueName: \"kubernetes.io/projected/93fdfdc7-e018-42f1-8a2d-03f87df6f493-kube-api-access-2wgrr\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.167270 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.167217 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.167339 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.167287 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.167925 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.167903 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.169631 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.169606 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.222161 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.222130 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgrr\" (UniqueName: \"kubernetes.io/projected/93fdfdc7-e018-42f1-8a2d-03f87df6f493-kube-api-access-2wgrr\") pod \"maas-api-revoke-sub-69969f78-nw5l8\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.329961 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.329874 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:01:58.696682 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.696649 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8"] Nov 03 14:01:58.697675 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:01:58.697647 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fdfdc7_e018_42f1_8a2d_03f87df6f493.slice/crio-b19c35b799aef9089f630265953037506cfe70c198ab89182f5943baf5f9f8db WatchSource:0}: Error finding container b19c35b799aef9089f630265953037506cfe70c198ab89182f5943baf5f9f8db: Status 404 returned error can't find the container with id b19c35b799aef9089f630265953037506cfe70c198ab89182f5943baf5f9f8db Nov 03 14:01:58.736929 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:58.736890 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" event={"ID":"93fdfdc7-e018-42f1-8a2d-03f87df6f493","Type":"ContainerStarted","Data":"b19c35b799aef9089f630265953037506cfe70c198ab89182f5943baf5f9f8db"} Nov 03 14:01:59.745575 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:59.745540 4924 generic.go:356] "Generic (PLEG): container finished" podID="93fdfdc7-e018-42f1-8a2d-03f87df6f493" containerID="33a5593016daa1f7ad3a67142005f8d5cc4ca01bf552d4aa1285a662597b40a4" exitCode=0 Nov 03 14:01:59.746000 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:01:59.745634 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" event={"ID":"93fdfdc7-e018-42f1-8a2d-03f87df6f493","Type":"ContainerDied","Data":"33a5593016daa1f7ad3a67142005f8d5cc4ca01bf552d4aa1285a662597b40a4"} Nov 03 14:02:00.913649 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:00.913623 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:02:00.998347 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:00.998306 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fdfdc7-e018-42f1-8a2d-03f87df6f493-kube-api-access-2wgrr\" (UniqueName: \"kubernetes.io/projected/93fdfdc7-e018-42f1-8a2d-03f87df6f493-kube-api-access-2wgrr\") pod \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " Nov 03 14:02:00.998545 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:00.998473 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-openshift-ca\") pod \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " Nov 03 14:02:00.998545 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:00.998507 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-xks-ca\") pod \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\" (UID: \"93fdfdc7-e018-42f1-8a2d-03f87df6f493\") " Nov 03 14:02:00.999330 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:00.999294 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-openshift-ca" pod "93fdfdc7-e018-42f1-8a2d-03f87df6f493" (UID: "93fdfdc7-e018-42f1-8a2d-03f87df6f493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:02:00.999787 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:00.999761 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:01.001281 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:01.001228 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fdfdc7-e018-42f1-8a2d-03f87df6f493-kube-api-access-2wgrr" pod "93fdfdc7-e018-42f1-8a2d-03f87df6f493" (UID: "93fdfdc7-e018-42f1-8a2d-03f87df6f493"). InnerVolumeSpecName "kube-api-access-2wgrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:02:01.010161 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:01.008524 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-xks-ca" pod "93fdfdc7-e018-42f1-8a2d-03f87df6f493" (UID: "93fdfdc7-e018-42f1-8a2d-03f87df6f493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:02:01.100445 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:01.100414 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fdfdc7-e018-42f1-8a2d-03f87df6f493-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:01.100445 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:01.100443 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wgrr\" (UniqueName: \"kubernetes.io/projected/93fdfdc7-e018-42f1-8a2d-03f87df6f493-kube-api-access-2wgrr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:01.760495 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:01.760461 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" Nov 03 14:02:01.760697 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:01.760491 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8" event={"ID":"93fdfdc7-e018-42f1-8a2d-03f87df6f493","Type":"ContainerDied","Data":"b19c35b799aef9089f630265953037506cfe70c198ab89182f5943baf5f9f8db"} Nov 03 14:02:01.760697 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:01.760584 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19c35b799aef9089f630265953037506cfe70c198ab89182f5943baf5f9f8db" Nov 03 14:02:03.292637 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:03.292608 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:08.337110 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:08.337011 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:11.774299 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:11.774264 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw"] Nov 03 14:02:11.797745 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:11.797709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:13.500862 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:13.500822 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:18.000744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.000707 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g"] Nov 03 14:02:18.001295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.001228 4924 cpu_manager.go:414] "containerMap: removing container" podUID="93fdfdc7-e018-42f1-8a2d-03f87df6f493" containerName="revoke-keys" Nov 03 14:02:18.001295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.001241 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fdfdc7-e018-42f1-8a2d-03f87df6f493" containerName="revoke-keys" Nov 03 14:02:18.001383 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.001348 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fdfdc7-e018-42f1-8a2d-03f87df6f493" containerName="revoke-keys" Nov 03 14:02:18.004814 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.004796 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.054904 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.054874 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c4e398-openshift-default-dockercfg-d4vg5\"" Nov 03 14:02:18.085100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085049 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ncz\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-kube-api-access-p7ncz\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.085260 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085157 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.085260 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085205 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.085260 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085224 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.085401 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085324 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1520ae8-820c-4f50-b823-770d7fa776f4-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.085532 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085506 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.085589 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085572 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.085645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085628 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.085690 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.085672 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.106243 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.106210 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g"] Nov 03 14:02:18.186443 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186410 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.186626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186458 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.186626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186573 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.186626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186617 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.186733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186663 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ncz\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-kube-api-access-p7ncz\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.186733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186692 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.186847 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186793 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.186902 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186845 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.186902 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186855 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.187004 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.186961 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1520ae8-820c-4f50-b823-770d7fa776f4-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.187080 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.187033 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.187244 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.187225 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.187310 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.187278 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.187592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.187570 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1520ae8-820c-4f50-b823-770d7fa776f4-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.188880 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.188859 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.189289 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.189271 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.241364 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.241335 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.241579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.241556 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ncz\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-kube-api-access-p7ncz\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.318632 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.318554 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:18.559363 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:02:18.559320 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:02:18.753841 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.753811 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g"] Nov 03 14:02:18.755217 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:02:18.755163 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1520ae8_820c_4f50_b823_770d7fa776f4.slice/crio-c6af66dae23bab4a1d277ba0e886206ac37248724e0640ddf797aa5a214f0558 WatchSource:0}: Error finding container c6af66dae23bab4a1d277ba0e886206ac37248724e0640ddf797aa5a214f0558: Status 404 returned error can't find the container with id c6af66dae23bab4a1d277ba0e886206ac37248724e0640ddf797aa5a214f0558 Nov 03 14:02:18.757823 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.757790 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:18.757901 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.757856 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:18.757901 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.757884 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:18.797502 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.797468 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:18.887687 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.887644 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" event={"ID":"a1520ae8-820c-4f50-b823-770d7fa776f4","Type":"ContainerStarted","Data":"826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca"} Nov 03 14:02:18.887851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:18.887713 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" event={"ID":"a1520ae8-820c-4f50-b823-770d7fa776f4","Type":"ContainerStarted","Data":"c6af66dae23bab4a1d277ba0e886206ac37248724e0640ddf797aa5a214f0558"} Nov 03 14:02:19.002708 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:19.002611 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podStartSLOduration=2.002594754 podStartE2EDuration="2.002594754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:18.999428943 +0000 UTC m=+3472.193832538" watchObservedRunningTime="2026-11-03 14:02:19.002594754 +0000 UTC m=+3472.196998348" Nov 03 14:02:19.318935 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:19.318854 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:19.320133 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:19.320115 4924 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 14:02:19.320226 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:19.320153 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 14:02:20.319570 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:20.319532 4924 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 14:02:20.319982 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:20.319596 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 14:02:21.319849 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:21.319813 4924 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 14:02:21.320310 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:21.319871 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 14:02:22.319548 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:22.319502 4924 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 14:02:22.319729 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:22.319564 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 14:02:23.319468 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:23.319421 4924 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 14:02:23.319999 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:23.319487 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 14:02:23.996979 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:23.996942 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:24.319420 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:24.319327 4924 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 14:02:24.319420 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:24.319392 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 14:02:25.324651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:25.324617 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:25.325056 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:25.325023 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:25.325714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:25.325692 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:02:26.798206 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:26.798157 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" podUID="fae1de5e-75c6-4914-a625-1eee2201cdbf" containerName="main" containerID="cri-o://79101288eaf6bc77de2432b0b09f86f0906972c8d1d03c4826a36b739b0076fe" gracePeriod=45 Nov 03 14:02:26.895871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:26.895833 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" podUID="fae1de5e-75c6-4914-a625-1eee2201cdbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": dial tcp 10.132.0.126:8000: connect: connection refused" Nov 03 14:02:26.969154 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:26.969116 4924 generic.go:356] "Generic (PLEG): container finished" podID="fae1de5e-75c6-4914-a625-1eee2201cdbf" containerID="79101288eaf6bc77de2432b0b09f86f0906972c8d1d03c4826a36b739b0076fe" exitCode=0 Nov 03 14:02:26.969323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:26.969196 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" event={"ID":"fae1de5e-75c6-4914-a625-1eee2201cdbf","Type":"ContainerDied","Data":"79101288eaf6bc77de2432b0b09f86f0906972c8d1d03c4826a36b739b0076fe"} Nov 03 14:02:27.069156 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.069127 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 14:02:27.210476 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.210433 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-dshm\") pod \"fae1de5e-75c6-4914-a625-1eee2201cdbf\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " Nov 03 14:02:27.210682 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.210500 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae1de5e-75c6-4914-a625-1eee2201cdbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/fae1de5e-75c6-4914-a625-1eee2201cdbf-tls-certs\") pod \"fae1de5e-75c6-4914-a625-1eee2201cdbf\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " Nov 03 14:02:27.210682 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.210659 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-model-cache\") pod \"fae1de5e-75c6-4914-a625-1eee2201cdbf\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " Nov 03 14:02:27.210816 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.210716 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae1de5e-75c6-4914-a625-1eee2201cdbf-kube-api-access-qwkkm\" (UniqueName: \"kubernetes.io/projected/fae1de5e-75c6-4914-a625-1eee2201cdbf-kube-api-access-qwkkm\") pod \"fae1de5e-75c6-4914-a625-1eee2201cdbf\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " Nov 03 14:02:27.210816 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.210773 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-home\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-home\") pod \"fae1de5e-75c6-4914-a625-1eee2201cdbf\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " Nov 03 14:02:27.210940 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.210830 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-tmp-dir\") pod \"fae1de5e-75c6-4914-a625-1eee2201cdbf\" (UID: \"fae1de5e-75c6-4914-a625-1eee2201cdbf\") " Nov 03 14:02:27.210998 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.210964 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-model-cache" pod "fae1de5e-75c6-4914-a625-1eee2201cdbf" (UID: "fae1de5e-75c6-4914-a625-1eee2201cdbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:02:27.211227 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.211202 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-home" pod "fae1de5e-75c6-4914-a625-1eee2201cdbf" (UID: "fae1de5e-75c6-4914-a625-1eee2201cdbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:02:27.211348 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.211278 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-tmp-dir" pod "fae1de5e-75c6-4914-a625-1eee2201cdbf" (UID: "fae1de5e-75c6-4914-a625-1eee2201cdbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:02:27.211483 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.211463 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:27.211566 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.211488 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:27.211566 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.211503 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:27.213061 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.213006 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-dshm" pod "fae1de5e-75c6-4914-a625-1eee2201cdbf" (UID: "fae1de5e-75c6-4914-a625-1eee2201cdbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:02:27.213264 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.213235 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae1de5e-75c6-4914-a625-1eee2201cdbf-kube-api-access-qwkkm" pod "fae1de5e-75c6-4914-a625-1eee2201cdbf" (UID: "fae1de5e-75c6-4914-a625-1eee2201cdbf"). InnerVolumeSpecName "kube-api-access-qwkkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:02:27.213352 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.213263 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae1de5e-75c6-4914-a625-1eee2201cdbf-tls-certs" pod "fae1de5e-75c6-4914-a625-1eee2201cdbf" (UID: "fae1de5e-75c6-4914-a625-1eee2201cdbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:02:27.313005 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.312925 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fae1de5e-75c6-4914-a625-1eee2201cdbf-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:27.313005 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.312955 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fae1de5e-75c6-4914-a625-1eee2201cdbf-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:27.313005 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.312965 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkkm\" (UniqueName: \"kubernetes.io/projected/fae1de5e-75c6-4914-a625-1eee2201cdbf-kube-api-access-qwkkm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:27.977445 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.977418 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" Nov 03 14:02:27.977445 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.977438 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw" event={"ID":"fae1de5e-75c6-4914-a625-1eee2201cdbf","Type":"ContainerDied","Data":"50eef8e74ffc2e138564092284ce53d6adfef7ba7d8af40184951bd0eaf21f0f"} Nov 03 14:02:27.977966 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:27.977487 4924 scope.go:122] "RemoveContainer" containerID="79101288eaf6bc77de2432b0b09f86f0906972c8d1d03c4826a36b739b0076fe" Nov 03 14:02:28.132346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:28.132305 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw"] Nov 03 14:02:28.179043 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:28.179009 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/auth-test-model-60261a-kserve-b7c7cb8b5-zvbcw"] Nov 03 14:02:28.951190 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:28.951153 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae1de5e-75c6-4914-a625-1eee2201cdbf" path="/var/lib/kubelet/pods/fae1de5e-75c6-4914-a625-1eee2201cdbf/volumes" Nov 03 14:02:29.173592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:29.173552 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:30.013397 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:30.013358 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:31.352417 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:31.352385 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:31.618037 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:31.617950 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:31.618272 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:31.618251 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c"] Nov 03 14:02:32.021222 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.021184 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h"] Nov 03 14:02:32.021811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.021789 4924 cpu_manager.go:414] "containerMap: removing container" podUID="fae1de5e-75c6-4914-a625-1eee2201cdbf" containerName="main" Nov 03 14:02:32.021918 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.021813 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae1de5e-75c6-4914-a625-1eee2201cdbf" containerName="main" Nov 03 14:02:32.021975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.021943 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae1de5e-75c6-4914-a625-1eee2201cdbf" containerName="main" Nov 03 14:02:32.025925 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.025899 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.081431 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.081392 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d5d415-openshift-default-dockercfg-p5fv7\"" Nov 03 14:02:32.082699 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.082657 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-socket\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.082852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.082727 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-podinfo\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.082852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.082756 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d8e5c4-5ba1-481e-9a0b-cec934989777-istiod-ca-cert\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.082852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.082808 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-token\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.082852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.082837 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-envoy\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.083136 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.082901 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-data\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.083136 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.082935 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2497\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-kube-api-access-f2497\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.083136 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.082994 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-credential-socket\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.083486 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.083146 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-certs\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.084695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.084670 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h"] Nov 03 14:02:32.108754 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.108718 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184119 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-certs\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184241 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-socket\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184281 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-podinfo\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184307 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d8e5c4-5ba1-481e-9a0b-cec934989777-istiod-ca-cert\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184352 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-token\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184378 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-envoy\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184431 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-data\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184461 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2497\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-kube-api-access-f2497\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184511 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-credential-socket\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184721 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-socket\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184917 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-credential-socket\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.188095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.184980 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-certs\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.193102 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.192106 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d8e5c4-5ba1-481e-9a0b-cec934989777-istiod-ca-cert\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.210094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.203817 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-envoy\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.210094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.208455 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-data\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.230410 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.220224 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-podinfo\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.250478 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.250429 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:32.255575 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.255539 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-token\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.256208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.256183 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2497\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-kube-api-access-f2497\") pod \"e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.360534 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.360444 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:32.790360 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.790330 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h"] Nov 03 14:02:32.790670 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:02:32.790646 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d8e5c4_5ba1_481e_9a0b_cec934989777.slice/crio-144a1a2dfd4d0148f3e579761d7189a3827c64a01559109a7493c609558f8248 WatchSource:0}: Error finding container 144a1a2dfd4d0148f3e579761d7189a3827c64a01559109a7493c609558f8248: Status 404 returned error can't find the container with id 144a1a2dfd4d0148f3e579761d7189a3827c64a01559109a7493c609558f8248 Nov 03 14:02:32.793371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.793336 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:32.793474 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.793414 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:32.793474 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:32.793460 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:33.016463 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:33.016425 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" event={"ID":"21d8e5c4-5ba1-481e-9a0b-cec934989777","Type":"ContainerStarted","Data":"607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682"} Nov 03 14:02:33.016653 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:33.016481 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" event={"ID":"21d8e5c4-5ba1-481e-9a0b-cec934989777","Type":"ContainerStarted","Data":"144a1a2dfd4d0148f3e579761d7189a3827c64a01559109a7493c609558f8248"} Nov 03 14:02:33.135627 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:33.135505 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" podStartSLOduration=2.135486525 podStartE2EDuration="2.135486525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:33.13048343 +0000 UTC m=+3486.324887024" watchObservedRunningTime="2026-11-03 14:02:33.135486525 +0000 UTC m=+3486.329890120" Nov 03 14:02:33.178839 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:33.178804 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:33.360697 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:33.360652 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:33.367173 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:33.367143 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:34.023788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:34.023757 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:34.024807 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:34.024786 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:02:34.526528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:34.526483 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:34.713247 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:34.713212 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:34.829214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:34.829134 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:36.131135 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:36.131092 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:36.249702 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:36.249669 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:36.377780 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:36.377741 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:39.422010 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:39.421960 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:39.598876 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:39.598841 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:40.154665 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:40.154630 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:40.352204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:40.352164 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:40.482276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:40.482228 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:44.407924 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:44.407886 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:44.587771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:44.587738 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:46.019734 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.019700 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:46.638804 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.638759 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" podUID="6d71c46c-cf44-4230-b70a-997a0a23e989" containerName="main" containerID="cri-o://f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c" gracePeriod=45 Nov 03 14:02:46.676276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.676239 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" podUID="6d71c46c-cf44-4230-b70a-997a0a23e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.128:8000/ready\": dial tcp 10.132.0.128:8000: connect: connection refused" Nov 03 14:02:46.923896 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.923864 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 14:02:46.994480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.994447 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-tmp-dir\") pod \"6d71c46c-cf44-4230-b70a-997a0a23e989\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " Nov 03 14:02:46.994659 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.994488 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-dshm\") pod \"6d71c46c-cf44-4230-b70a-997a0a23e989\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " Nov 03 14:02:46.994659 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.994543 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d71c46c-cf44-4230-b70a-997a0a23e989-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d71c46c-cf44-4230-b70a-997a0a23e989-tls-certs\") pod \"6d71c46c-cf44-4230-b70a-997a0a23e989\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " Nov 03 14:02:46.994659 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.994589 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d71c46c-cf44-4230-b70a-997a0a23e989-kube-api-access-dwbms\" (UniqueName: \"kubernetes.io/projected/6d71c46c-cf44-4230-b70a-997a0a23e989-kube-api-access-dwbms\") pod \"6d71c46c-cf44-4230-b70a-997a0a23e989\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " Nov 03 14:02:46.994837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.994659 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-model-cache\") pod \"6d71c46c-cf44-4230-b70a-997a0a23e989\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " Nov 03 14:02:46.994837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.994714 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-tmp-dir" pod "6d71c46c-cf44-4230-b70a-997a0a23e989" (UID: "6d71c46c-cf44-4230-b70a-997a0a23e989"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:02:46.994837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.994764 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-home\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-home\") pod \"6d71c46c-cf44-4230-b70a-997a0a23e989\" (UID: \"6d71c46c-cf44-4230-b70a-997a0a23e989\") " Nov 03 14:02:46.995010 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.994955 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-model-cache" pod "6d71c46c-cf44-4230-b70a-997a0a23e989" (UID: "6d71c46c-cf44-4230-b70a-997a0a23e989"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:02:46.995136 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.995112 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-home" pod "6d71c46c-cf44-4230-b70a-997a0a23e989" (UID: "6d71c46c-cf44-4230-b70a-997a0a23e989"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:02:46.995280 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.995258 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:46.995280 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.995280 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:46.995451 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.995295 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:46.997122 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.997096 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-dshm" pod "6d71c46c-cf44-4230-b70a-997a0a23e989" (UID: "6d71c46c-cf44-4230-b70a-997a0a23e989"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:02:46.997290 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.997259 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d71c46c-cf44-4230-b70a-997a0a23e989-kube-api-access-dwbms" pod "6d71c46c-cf44-4230-b70a-997a0a23e989" (UID: "6d71c46c-cf44-4230-b70a-997a0a23e989"). InnerVolumeSpecName "kube-api-access-dwbms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:02:46.997433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:46.997335 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d71c46c-cf44-4230-b70a-997a0a23e989-tls-certs" pod "6d71c46c-cf44-4230-b70a-997a0a23e989" (UID: "6d71c46c-cf44-4230-b70a-997a0a23e989"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:02:47.096854 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.096816 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d71c46c-cf44-4230-b70a-997a0a23e989-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:47.096854 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.096845 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d71c46c-cf44-4230-b70a-997a0a23e989-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:47.096854 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.096855 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbms\" (UniqueName: \"kubernetes.io/projected/6d71c46c-cf44-4230-b70a-997a0a23e989-kube-api-access-dwbms\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:47.134771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.134738 4924 generic.go:356] "Generic (PLEG): container finished" podID="6d71c46c-cf44-4230-b70a-997a0a23e989" containerID="f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c" exitCode=0 Nov 03 14:02:47.134932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.134777 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" Nov 03 14:02:47.134932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.134820 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" event={"ID":"6d71c46c-cf44-4230-b70a-997a0a23e989","Type":"ContainerDied","Data":"f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c"} Nov 03 14:02:47.134932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.134866 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c" event={"ID":"6d71c46c-cf44-4230-b70a-997a0a23e989","Type":"ContainerDied","Data":"e4d4c4abb76bb0e6e79ca3182b4e26be2c170839a51261daa6ccdd5880398ef0"} Nov 03 14:02:47.134932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.134883 4924 scope.go:122] "RemoveContainer" containerID="f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c" Nov 03 14:02:47.145292 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.145273 4924 scope.go:122] "RemoveContainer" containerID="f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c" Nov 03 14:02:47.145565 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:02:47.145542 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c\": container with ID starting with f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c not found: ID does not exist" containerID="f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c" Nov 03 14:02:47.145612 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.145574 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c"} err="failed to get container status \"f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c\": rpc error: code = NotFound desc = could not find container \"f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c\": container with ID starting with f84d598c1a1b20a283a5e4befcbd80a9fc2f9b20f13a1b8ec121f1eca6aa7c5c not found: ID does not exist" Nov 03 14:02:47.291999 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.291967 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c"] Nov 03 14:02:47.294981 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:47.294958 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/auth-test-model-367381-kserve-7b7b4db645-48z9c"] Nov 03 14:02:48.559753 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:02:48.559716 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:02:48.951531 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:48.951493 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d71c46c-cf44-4230-b70a-997a0a23e989" path="/var/lib/kubelet/pods/6d71c46c-cf44-4230-b70a-997a0a23e989/volumes" Nov 03 14:02:51.457650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:51.457617 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:56.805237 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.805199 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4"] Nov 03 14:02:56.805980 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.805962 4924 cpu_manager.go:414] "containerMap: removing container" podUID="6d71c46c-cf44-4230-b70a-997a0a23e989" containerName="main" Nov 03 14:02:56.806025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.805982 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d71c46c-cf44-4230-b70a-997a0a23e989" containerName="main" Nov 03 14:02:56.806126 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.806114 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d71c46c-cf44-4230-b70a-997a0a23e989" containerName="main" Nov 03 14:02:56.811300 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.811280 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:56.863113 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.863055 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4"] Nov 03 14:02:56.904908 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.904879 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-58f896-serving-cert\"" Nov 03 14:02:56.912112 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.912082 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-metrics-certs\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:56.912236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.912158 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztl2l\" (UniqueName: \"kubernetes.io/projected/d26885da-e7a5-41f7-a869-47930fa5f1a0-kube-api-access-ztl2l\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:56.912283 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.912240 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26885da-e7a5-41f7-a869-47930fa5f1a0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:56.912339 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:56.912322 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.013411 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.013362 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-metrics-certs\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.013605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.013418 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztl2l\" (UniqueName: \"kubernetes.io/projected/d26885da-e7a5-41f7-a869-47930fa5f1a0-kube-api-access-ztl2l\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.013605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.013525 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26885da-e7a5-41f7-a869-47930fa5f1a0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.013605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.013553 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.013792 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:02:57.013655 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-serving-cert: secret "maas-api-e2e-extmod-a-58f896-serving-cert" not found Nov 03 14:02:57.013792 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:02:57.013715 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls podName:d26885da-e7a5-41f7-a869-47930fa5f1a0 nodeName:}" failed. No retries permitted until 2026-11-03 14:02:57.513693281 +0000 UTC m=+3510.708096868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls") pod "maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" (UID: "d26885da-e7a5-41f7-a869-47930fa5f1a0") : secret "maas-api-e2e-extmod-a-58f896-serving-cert" not found Nov 03 14:02:57.014201 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.014175 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26885da-e7a5-41f7-a869-47930fa5f1a0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.015846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.015825 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-metrics-certs\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.029297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.029236 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:57.070788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.070713 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztl2l\" (UniqueName: \"kubernetes.io/projected/d26885da-e7a5-41f7-a869-47930fa5f1a0-kube-api-access-ztl2l\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.174403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.174363 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:57.298876 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.298840 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:02:57.518689 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.518650 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.521246 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.521219 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:57.724702 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:57.724660 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:58.101278 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:58.101251 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4"] Nov 03 14:02:58.103052 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:02:58.103016 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26885da_e7a5_41f7_a869_47930fa5f1a0.slice/crio-16d1c117b79dbe279e016a3719aa9b3d68085dce0b500b911aea973ee708e8ff WatchSource:0}: Error finding container 16d1c117b79dbe279e016a3719aa9b3d68085dce0b500b911aea973ee708e8ff: Status 404 returned error can't find the container with id 16d1c117b79dbe279e016a3719aa9b3d68085dce0b500b911aea973ee708e8ff Nov 03 14:02:58.211714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:58.211672 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" event={"ID":"d26885da-e7a5-41f7-a869-47930fa5f1a0","Type":"ContainerStarted","Data":"16d1c117b79dbe279e016a3719aa9b3d68085dce0b500b911aea973ee708e8ff"} Nov 03 14:02:59.223582 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:59.223435 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" event={"ID":"d26885da-e7a5-41f7-a869-47930fa5f1a0","Type":"ContainerStarted","Data":"dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c"} Nov 03 14:02:59.224682 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:59.224615 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:02:59.340359 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:02:59.338842 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" podStartSLOduration=2.361815678 podStartE2EDuration="3.338820562s" totalImagesPullingTime="977.004884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:59.334311722 +0000 UTC m=+3512.528715319" watchObservedRunningTime="2026-11-03 14:02:59.338820562 +0000 UTC m=+3512.533224157" Nov 03 14:03:01.073981 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:01.073949 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:02.426028 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:02.425991 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:02.835596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:02.835511 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:03.594294 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:03.594241 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:03.761921 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:03.761887 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:05.243748 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:05.243715 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:03:07.587943 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:07.587904 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:07.815903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:07.815852 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:07.948252 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:07.948218 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:08.361313 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:08.361215 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:09.073106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:09.073046 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns"] Nov 03 14:03:09.117339 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:09.117303 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f124edb-4jmns"] Nov 03 14:03:10.953002 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:10.952963 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbe0bfc-e2aa-4520-8609-b3209c038fe1" path="/var/lib/kubelet/pods/2cbe0bfc-e2aa-4520-8609-b3209c038fe1/volumes" Nov 03 14:03:12.244969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:12.244933 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:12.426677 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:12.426640 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:13.771323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:13.771274 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:18.558856 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:03:18.558812 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:03:19.051112 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:19.051052 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:20.047691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:20.047645 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr"] Nov 03 14:03:20.093594 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:20.093557 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc96c114-5q8mr"] Nov 03 14:03:20.950998 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:20.950955 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47db33e4-c501-45b8-b3f2-494660c9aefa" path="/var/lib/kubelet/pods/47db33e4-c501-45b8-b3f2-494660c9aefa/volumes" Nov 03 14:03:24.087301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:24.087264 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:29.095839 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:29.095805 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:34.158318 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:34.158279 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:36.462468 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:36.462434 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:39.180280 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:39.180241 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:44.278215 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:44.278174 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:48.564576 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:03:48.564533 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:03:49.375667 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:49.375634 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:50.045658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:50.045617 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp"] Nov 03 14:03:50.091179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:50.091145 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd726f0-vcwwp"] Nov 03 14:03:50.951615 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:50.951583 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b106ac-797e-43f6-bb2e-7e8a83fd8e19" path="/var/lib/kubelet/pods/64b106ac-797e-43f6-bb2e-7e8a83fd8e19/volumes" Nov 03 14:03:52.542194 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:52.542158 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7606-c8ec03"] Nov 03 14:03:52.546164 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:52.546146 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:52.578648 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:52.578614 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2ft\" (UniqueName: \"kubernetes.io/projected/72750a21-7ab1-48e5-9e16-fa4cf78003a7-kube-api-access-cv2ft\") pod \"test-curl-7606-c8ec03\" (UID: \"72750a21-7ab1-48e5-9e16-fa4cf78003a7\") " pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:52.597212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:52.597187 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7606-c8ec03"] Nov 03 14:03:52.639503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:52.639473 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kmbtk\"" Nov 03 14:03:52.679890 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:52.679854 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2ft\" (UniqueName: \"kubernetes.io/projected/72750a21-7ab1-48e5-9e16-fa4cf78003a7-kube-api-access-cv2ft\") pod \"test-curl-7606-c8ec03\" (UID: \"72750a21-7ab1-48e5-9e16-fa4cf78003a7\") " pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:52.733009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:52.732978 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2ft\" (UniqueName: \"kubernetes.io/projected/72750a21-7ab1-48e5-9e16-fa4cf78003a7-kube-api-access-cv2ft\") pod \"test-curl-7606-c8ec03\" (UID: \"72750a21-7ab1-48e5-9e16-fa4cf78003a7\") " pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:52.859212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:52.859132 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:53.226303 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:53.226270 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7606-c8ec03"] Nov 03 14:03:53.226929 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:03:53.226899 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72750a21_7ab1_48e5_9e16_fa4cf78003a7.slice/crio-2fc521a5b51969dd2329ac12bdd99c39bf1948c226b0f23aa063fc9a019bf4cd WatchSource:0}: Error finding container 2fc521a5b51969dd2329ac12bdd99c39bf1948c226b0f23aa063fc9a019bf4cd: Status 404 returned error can't find the container with id 2fc521a5b51969dd2329ac12bdd99c39bf1948c226b0f23aa063fc9a019bf4cd Nov 03 14:03:53.646749 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:53.646710 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7606-c8ec03" event={"ID":"72750a21-7ab1-48e5-9e16-fa4cf78003a7","Type":"ContainerStarted","Data":"0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0"} Nov 03 14:03:53.646749 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:53.646757 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7606-c8ec03" event={"ID":"72750a21-7ab1-48e5-9e16-fa4cf78003a7","Type":"ContainerStarted","Data":"2fc521a5b51969dd2329ac12bdd99c39bf1948c226b0f23aa063fc9a019bf4cd"} Nov 03 14:03:53.753381 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:53.753325 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7606-c8ec03" podStartSLOduration=1.753310347 podStartE2EDuration="1.753310347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:53.751656964 +0000 UTC m=+3566.946060555" watchObservedRunningTime="2026-11-03 14:03:53.753310347 +0000 UTC m=+3566.947713941" Nov 03 14:03:54.551723 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:54.551687 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:03:55.141200 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:55.141162 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7606-c8ec03"] Nov 03 14:03:55.146890 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:55.146858 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7606-c8ec03"] Nov 03 14:03:55.661328 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:55.661260 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7606-c8ec03" podUID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" containerName="test-curl-7606-c8ec03" containerID="cri-o://0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0" gracePeriod=2 Nov 03 14:03:55.708955 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:03:55.708915 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7606-c8ec03\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:57.001262 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:03:57.001222 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7606-c8ec03\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:57.898763 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:57.898733 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:58.041706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.041609 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72750a21-7ab1-48e5-9e16-fa4cf78003a7-kube-api-access-cv2ft\" (UniqueName: \"kubernetes.io/projected/72750a21-7ab1-48e5-9e16-fa4cf78003a7-kube-api-access-cv2ft\") pod \"72750a21-7ab1-48e5-9e16-fa4cf78003a7\" (UID: \"72750a21-7ab1-48e5-9e16-fa4cf78003a7\") " Nov 03 14:03:58.044414 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.044377 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72750a21-7ab1-48e5-9e16-fa4cf78003a7-kube-api-access-cv2ft" pod "72750a21-7ab1-48e5-9e16-fa4cf78003a7" (UID: "72750a21-7ab1-48e5-9e16-fa4cf78003a7"). InnerVolumeSpecName "kube-api-access-cv2ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:03:58.143659 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.143618 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2ft\" (UniqueName: \"kubernetes.io/projected/72750a21-7ab1-48e5-9e16-fa4cf78003a7-kube-api-access-cv2ft\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:58.680739 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.680700 4924 generic.go:356] "Generic (PLEG): container finished" podID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" containerID="0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0" exitCode=137 Nov 03 14:03:58.680934 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.680751 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:58.680934 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.680801 4924 scope.go:122] "RemoveContainer" containerID="0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0" Nov 03 14:03:58.690993 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.690973 4924 scope.go:122] "RemoveContainer" containerID="0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0" Nov 03 14:03:58.691282 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:03:58.691264 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0\": container with ID starting with 0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0 not found: ID does not exist" containerID="0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0" Nov 03 14:03:58.691344 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.691292 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0"} err="failed to get container status \"0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0\": rpc error: code = NotFound desc = could not find container \"0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0\": container with ID starting with 0faa8ab7496d8b1cafe33e8fe74e0e0793a50b34ab131e1aabdbf50a656365e0 not found: ID does not exist" Nov 03 14:03:58.737274 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:03:58.737235 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7606-c8ec03\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" pod="opendatahub/test-curl-7606-c8ec03" Nov 03 14:03:58.950902 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:58.950823 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" path="/var/lib/kubelet/pods/72750a21-7ab1-48e5-9e16-fa4cf78003a7/volumes" Nov 03 14:03:59.595455 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:03:59.595411 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:01.045365 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:01.045330 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9"] Nov 03 14:04:01.092048 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:01.092014 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4274dd0-msdc9"] Nov 03 14:04:02.951800 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:02.951756 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa9965c-54b4-4492-9dcf-ba27e74c028e" path="/var/lib/kubelet/pods/6aa9965c-54b4-4492-9dcf-ba27e74c028e/volumes" Nov 03 14:04:04.319361 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.319325 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d"] Nov 03 14:04:04.319830 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.319816 4924 cpu_manager.go:414] "containerMap: removing container" podUID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" containerName="test-curl-7606-c8ec03" Nov 03 14:04:04.319830 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.319829 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" containerName="test-curl-7606-c8ec03" Nov 03 14:04:04.319947 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.319936 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="72750a21-7ab1-48e5-9e16-fa4cf78003a7" containerName="test-curl-7606-c8ec03" Nov 03 14:04:04.323346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.323323 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.373589 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.373553 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d5d415-serving-cert\"" Nov 03 14:04:04.416065 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.416020 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp87l\" (UniqueName: \"kubernetes.io/projected/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-kube-api-access-jp87l\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.416279 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.416190 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.416279 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.416225 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.416402 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.416378 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.423850 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.423821 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d"] Nov 03 14:04:04.507900 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.507865 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:04.517043 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.517017 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.517171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.517154 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp87l\" (UniqueName: \"kubernetes.io/projected/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-kube-api-access-jp87l\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.517233 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.517217 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.517274 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.517253 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.517417 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:04:04.517402 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-serving-cert: secret "maas-api-e2e-extmod-b-d5d415-serving-cert" not found Nov 03 14:04:04.517486 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:04:04.517476 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls podName:ea0f9151-ad2a-4ac2-b667-f58b1ec98e88 nodeName:}" failed. No retries permitted until 2026-11-03 14:04:05.017457603 +0000 UTC m=+3578.211861181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls") pod "maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" (UID: "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88") : secret "maas-api-e2e-extmod-b-d5d415-serving-cert" not found Nov 03 14:04:04.517767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.517751 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.519635 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.519617 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-metrics-certs\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.571833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.571756 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp87l\" (UniqueName: \"kubernetes.io/projected/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-kube-api-access-jp87l\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:04.670442 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.670399 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:04.888346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:04.888260 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:05.023475 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:05.023439 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:05.025895 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:05.025872 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:05.239605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:05.239570 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:05.616359 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:05.616325 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d"] Nov 03 14:04:05.616692 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:04:05.616390 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0f9151_ad2a_4ac2_b667_f58b1ec98e88.slice/crio-711ec7b4a425c23fa84f9ec9b0dbfebc360ea6ca6c102e876817d825a76963da WatchSource:0}: Error finding container 711ec7b4a425c23fa84f9ec9b0dbfebc360ea6ca6c102e876817d825a76963da: Status 404 returned error can't find the container with id 711ec7b4a425c23fa84f9ec9b0dbfebc360ea6ca6c102e876817d825a76963da Nov 03 14:04:05.727378 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:05.727336 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" event={"ID":"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88","Type":"ContainerStarted","Data":"711ec7b4a425c23fa84f9ec9b0dbfebc360ea6ca6c102e876817d825a76963da"} Nov 03 14:04:06.745728 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:06.745677 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" event={"ID":"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88","Type":"ContainerStarted","Data":"b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda"} Nov 03 14:04:06.746155 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:06.745920 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:06.857189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:06.857083 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" podStartSLOduration=1.915077293 podStartE2EDuration="2.857059186s" totalImagesPullingTime="941.981893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:06.853708402 +0000 UTC m=+3580.048111997" watchObservedRunningTime="2026-11-03 14:04:06.857059186 +0000 UTC m=+3580.051462830" Nov 03 14:04:09.762557 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:09.762509 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:12.757307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:12.757274 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:04:14.619007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.618964 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv"] Nov 03 14:04:14.623253 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.623229 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.639126 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.639097 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a180bf-eeb3-48af-bbc3-26518c39de5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.639293 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.639155 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-metrics-certs\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.639357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.639310 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnnzk\" (UniqueName: \"kubernetes.io/projected/a6a180bf-eeb3-48af-bbc3-26518c39de5f-kube-api-access-nnnzk\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.639413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.639366 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.674216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.674165 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6ebe57-serving-cert\"" Nov 03 14:04:14.724357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.724314 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv"] Nov 03 14:04:14.740325 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.740289 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.740515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.740369 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a180bf-eeb3-48af-bbc3-26518c39de5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.740515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.740438 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-metrics-certs\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.740515 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:04:14.740446 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-serving-cert: secret "maas-api-e2e-extmod-a-6ebe57-serving-cert" not found Nov 03 14:04:14.740657 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:04:14.740522 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls podName:a6a180bf-eeb3-48af-bbc3-26518c39de5f nodeName:}" failed. No retries permitted until 2026-11-03 14:04:15.240500354 +0000 UTC m=+3588.434903940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls") pod "maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" (UID: "a6a180bf-eeb3-48af-bbc3-26518c39de5f") : secret "maas-api-e2e-extmod-a-6ebe57-serving-cert" not found Nov 03 14:04:14.740657 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.740635 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnnzk\" (UniqueName: \"kubernetes.io/projected/a6a180bf-eeb3-48af-bbc3-26518c39de5f-kube-api-access-nnnzk\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.741717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.741688 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a180bf-eeb3-48af-bbc3-26518c39de5f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.743350 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.743305 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-metrics-certs\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.794984 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.794952 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnnzk\" (UniqueName: \"kubernetes.io/projected/a6a180bf-eeb3-48af-bbc3-26518c39de5f-kube-api-access-nnnzk\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:14.798286 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.798256 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:14.961878 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:14.961842 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:15.093723 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:15.093682 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:15.248112 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:15.247995 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:15.250729 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:15.250702 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:15.536694 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:15.536606 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:15.913109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:15.913062 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv"] Nov 03 14:04:15.914367 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:04:15.914339 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a180bf_eeb3_48af_bbc3_26518c39de5f.slice/crio-706951343cc31f74f0b73e527c7a8d48c229c074e1b9b4361a3876083faf99bf WatchSource:0}: Error finding container 706951343cc31f74f0b73e527c7a8d48c229c074e1b9b4361a3876083faf99bf: Status 404 returned error can't find the container with id 706951343cc31f74f0b73e527c7a8d48c229c074e1b9b4361a3876083faf99bf Nov 03 14:04:16.830581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:16.830539 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" event={"ID":"a6a180bf-eeb3-48af-bbc3-26518c39de5f","Type":"ContainerStarted","Data":"706951343cc31f74f0b73e527c7a8d48c229c074e1b9b4361a3876083faf99bf"} Nov 03 14:04:17.839170 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:17.839134 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" event={"ID":"a6a180bf-eeb3-48af-bbc3-26518c39de5f","Type":"ContainerStarted","Data":"37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb"} Nov 03 14:04:17.839638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:17.839358 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:17.952568 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:17.952505 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" podStartSLOduration=3.110101189 podStartE2EDuration="3.952486571s" totalImagesPullingTime="842.385382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:17.948466019 +0000 UTC m=+3591.142869616" watchObservedRunningTime="2026-11-03 14:04:17.952486571 +0000 UTC m=+3591.146890166" Nov 03 14:04:18.608252 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:04:18.608204 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:04:19.963024 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:19.962979 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:20.079723 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:20.079690 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:20.196643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:20.196607 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6"] Nov 03 14:04:20.202529 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:20.202493 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:23.851846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:23.851810 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:04:25.226661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:25.226619 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:27.542273 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:27.542243 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:04:27.552600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:27.552574 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:04:28.056185 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:28.056156 4924 scope.go:122] "RemoveContainer" containerID="4c488fa9ca056f00f1e100d56fa649699f304bee9841317418de000eacebd206" Nov 03 14:04:28.067868 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:28.067840 4924 scope.go:122] "RemoveContainer" containerID="dc84cc2b98eccf14ca23fa455789c60d2ee48e854f7cf46b49383991bdf9267b" Nov 03 14:04:29.399682 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:29.399648 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:30.279214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:30.279181 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:35.142972 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.142922 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:35.219225 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.219179 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" podUID="29cca271-b0b2-4925-a387-cd61c4c507f9" containerName="main" containerID="cri-o://a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c" gracePeriod=45 Nov 03 14:04:35.470898 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.470848 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:35.676506 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.676465 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 14:04:35.817722 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.817645 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-model-cache\") pod \"29cca271-b0b2-4925-a387-cd61c4c507f9\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " Nov 03 14:04:35.817722 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.817687 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29cca271-b0b2-4925-a387-cd61c4c507f9-kube-api-access-jl7zz\" (UniqueName: \"kubernetes.io/projected/29cca271-b0b2-4925-a387-cd61c4c507f9-kube-api-access-jl7zz\") pod \"29cca271-b0b2-4925-a387-cd61c4c507f9\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " Nov 03 14:04:35.817722 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.817719 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-home\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-home\") pod \"29cca271-b0b2-4925-a387-cd61c4c507f9\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " Nov 03 14:04:35.818014 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.817761 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-dshm\") pod \"29cca271-b0b2-4925-a387-cd61c4c507f9\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " Nov 03 14:04:35.818014 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.817810 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29cca271-b0b2-4925-a387-cd61c4c507f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/29cca271-b0b2-4925-a387-cd61c4c507f9-tls-certs\") pod \"29cca271-b0b2-4925-a387-cd61c4c507f9\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " Nov 03 14:04:35.818014 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.817932 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-tmp-dir\") pod \"29cca271-b0b2-4925-a387-cd61c4c507f9\" (UID: \"29cca271-b0b2-4925-a387-cd61c4c507f9\") " Nov 03 14:04:35.818014 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.817971 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-model-cache" pod "29cca271-b0b2-4925-a387-cd61c4c507f9" (UID: "29cca271-b0b2-4925-a387-cd61c4c507f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:04:35.818301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.818277 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-tmp-dir" pod "29cca271-b0b2-4925-a387-cd61c4c507f9" (UID: "29cca271-b0b2-4925-a387-cd61c4c507f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:04:35.818757 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.818626 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-home" pod "29cca271-b0b2-4925-a387-cd61c4c507f9" (UID: "29cca271-b0b2-4925-a387-cd61c4c507f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:04:35.818872 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.818765 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:35.818872 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.818783 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:35.818872 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.818797 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:35.820779 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.820749 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-dshm" pod "29cca271-b0b2-4925-a387-cd61c4c507f9" (UID: "29cca271-b0b2-4925-a387-cd61c4c507f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:04:35.820779 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.820755 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cca271-b0b2-4925-a387-cd61c4c507f9-tls-certs" pod "29cca271-b0b2-4925-a387-cd61c4c507f9" (UID: "29cca271-b0b2-4925-a387-cd61c4c507f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:04:35.820912 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.820764 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cca271-b0b2-4925-a387-cd61c4c507f9-kube-api-access-jl7zz" pod "29cca271-b0b2-4925-a387-cd61c4c507f9" (UID: "29cca271-b0b2-4925-a387-cd61c4c507f9"). InnerVolumeSpecName "kube-api-access-jl7zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:04:35.919946 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.919902 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cca271-b0b2-4925-a387-cd61c4c507f9-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:35.919946 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.919944 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cca271-b0b2-4925-a387-cd61c4c507f9-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:35.920237 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:35.919963 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7zz\" (UniqueName: \"kubernetes.io/projected/29cca271-b0b2-4925-a387-cd61c4c507f9-kube-api-access-jl7zz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:36.006434 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.006397 4924 generic.go:356] "Generic (PLEG): container finished" podID="29cca271-b0b2-4925-a387-cd61c4c507f9" containerID="a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c" exitCode=0 Nov 03 14:04:36.006633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.006440 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" Nov 03 14:04:36.006633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.006484 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" event={"ID":"29cca271-b0b2-4925-a387-cd61c4c507f9","Type":"ContainerDied","Data":"a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c"} Nov 03 14:04:36.006633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.006527 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6" event={"ID":"29cca271-b0b2-4925-a387-cd61c4c507f9","Type":"ContainerDied","Data":"1bdd789dca5d48be0c1795bb7652fe8a383d48e0f309349c2b2ff799f04bf7da"} Nov 03 14:04:36.006633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.006544 4924 scope.go:122] "RemoveContainer" containerID="a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c" Nov 03 14:04:36.017657 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.017634 4924 scope.go:122] "RemoveContainer" containerID="a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c" Nov 03 14:04:36.017956 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:04:36.017933 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c\": container with ID starting with a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c not found: ID does not exist" containerID="a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c" Nov 03 14:04:36.018037 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.017968 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c"} err="failed to get container status \"a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c\": rpc error: code = NotFound desc = could not find container \"a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c\": container with ID starting with a20ddc82810fba8e2b2b49d954553ea54c887c4b85436d0442882e5fe5861c4c not found: ID does not exist" Nov 03 14:04:36.166430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.166397 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6"] Nov 03 14:04:36.169282 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.169254 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5ae63d/auth-test-model-5ae63d-kserve-75b4656db9-tx9j6"] Nov 03 14:04:36.284807 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.284773 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:36.406244 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.406192 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:36.790735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.790697 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:36.955027 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:36.954991 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29cca271-b0b2-4925-a387-cd61c4c507f9" path="/var/lib/kubelet/pods/29cca271-b0b2-4925-a387-cd61c4c507f9/volumes" Nov 03 14:04:39.890576 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:39.890527 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:39.903711 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:39.903677 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79"] Nov 03 14:04:40.055146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:40.054980 4924 generic.go:356] "Generic (PLEG): container finished" podID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerID="607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682" exitCode=0 Nov 03 14:04:40.055146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:40.055051 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" event={"ID":"21d8e5c4-5ba1-481e-9a0b-cec934989777","Type":"ContainerDied","Data":"607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682"} Nov 03 14:04:40.055823 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:40.055799 4924 scope.go:122] "RemoveContainer" containerID="607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682" Nov 03 14:04:40.058738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:40.058694 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:04:40.058974 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:40.058940 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:04:40.059284 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:40.059004 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:04:40.487989 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:40.487947 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:40.609623 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:40.609586 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:41.111474 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.111436 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" event={"ID":"21d8e5c4-5ba1-481e-9a0b-cec934989777","Type":"ContainerStarted","Data":"da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3"} Nov 03 14:04:41.147474 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.147436 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5"] Nov 03 14:04:41.160772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.160737 4924 cpu_manager.go:414] "containerMap: removing container" podUID="29cca271-b0b2-4925-a387-cd61c4c507f9" containerName="main" Nov 03 14:04:41.161022 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.161003 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29cca271-b0b2-4925-a387-cd61c4c507f9" containerName="main" Nov 03 14:04:41.161524 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.161503 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="29cca271-b0b2-4925-a387-cd61c4c507f9" containerName="main" Nov 03 14:04:41.183457 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.183373 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.200469 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200425 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istiod-ca-cert\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.200636 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200485 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-certs\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.200636 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200528 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-socket\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.200636 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200569 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-data\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.200636 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200599 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-token\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.200858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200711 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-podinfo\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.200858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200765 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwrs\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-kube-api-access-xkwrs\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.200858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200853 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-credential-socket\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.201035 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.200881 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-envoy\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.215408 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.215373 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5"] Nov 03 14:04:41.262402 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.262329 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:41.282499 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.282346 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-aff1d2-openshift-default-dockercfg-jh2bk\"" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.301776 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-podinfo\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.301852 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwrs\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-kube-api-access-xkwrs\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.301940 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-credential-socket\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.301974 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-envoy\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.303134 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istiod-ca-cert\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.303199 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-certs\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.303255 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-socket\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.303313 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-data\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.303351 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-token\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304962 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.304526 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istiod-ca-cert\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.304962 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.304792 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-certs\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.305107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.305048 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-socket\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.305347 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.305320 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-data\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.305732 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.305691 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-credential-socket\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.308954 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.308145 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-envoy\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.310025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.309809 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-podinfo\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.368307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.368207 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:04:41.369383 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.369349 4924 patch_prober.go:29] interesting pod/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:04:41.375231 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.375176 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 14:04:41.384124 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.381957 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwrs\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-kube-api-access-xkwrs\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.389097 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.386756 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-token\") pod \"e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:41.399719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.399537 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:41.583115 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.579521 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:41.583115 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:41.581468 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:42.081257 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:42.079812 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5"] Nov 03 14:04:42.087965 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:42.087451 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:04:42.087965 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:42.087521 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:04:42.087965 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:42.087551 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:04:42.180053 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:42.180005 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" event={"ID":"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd","Type":"ContainerStarted","Data":"05f931d4c1e1dc86741367fcf3472411313017a258092eb7e1ec53379ba29535"} Nov 03 14:04:42.367557 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:42.367466 4924 patch_prober.go:29] interesting pod/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:04:42.367839 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:42.367809 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 14:04:43.188771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:43.188727 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" event={"ID":"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd","Type":"ContainerStarted","Data":"c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050"} Nov 03 14:04:43.360970 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:43.360930 4924 patch_prober.go:29] interesting pod/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:04:43.361208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:43.361004 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 14:04:43.386363 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:43.386303 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" podStartSLOduration=2.3862895650000002 podStartE2EDuration="2.386289565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:43.384289846 +0000 UTC m=+3616.578693440" watchObservedRunningTime="2026-11-03 14:04:43.386289565 +0000 UTC m=+3616.580693159" Nov 03 14:04:43.582709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:43.582436 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:43.590509 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:43.590469 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:44.195725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:44.195691 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:44.196634 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:44.196612 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:04:44.368718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:44.361462 4924 patch_prober.go:29] interesting pod/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:04:44.368718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:44.361524 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 14:04:45.361317 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:45.361283 4924 patch_prober.go:29] interesting pod/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:04:45.361794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:45.361767 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 14:04:45.656754 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:45.656540 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:45.816497 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:45.816451 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:45.950105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:45.949994 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:46.361835 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:46.361735 4924 patch_prober.go:29] interesting pod/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:04:46.361835 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:46.361801 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 14:04:46.419506 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:46.419470 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:46.622709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:46.622634 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:46.785902 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:46.785866 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:47.365816 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:47.365783 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:04:47.366244 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:47.366113 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:04:47.366916 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:47.366899 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:04:48.559377 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:04:48.559342 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:04:51.405108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:51.405061 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:51.549130 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:51.549028 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:51.667484 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:51.667453 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:04:54.923788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:54.923750 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" podUID="206eeab5-6be2-4392-b95d-4961a5aa1d43" containerName="main" containerID="cri-o://ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5" gracePeriod=45 Nov 03 14:04:55.190939 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.190875 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 14:04:55.275480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.275445 4924 generic.go:356] "Generic (PLEG): container finished" podID="206eeab5-6be2-4392-b95d-4961a5aa1d43" containerID="ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5" exitCode=0 Nov 03 14:04:55.275643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.275492 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" Nov 03 14:04:55.275643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.275527 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" event={"ID":"206eeab5-6be2-4392-b95d-4961a5aa1d43","Type":"ContainerDied","Data":"ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5"} Nov 03 14:04:55.275643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.275578 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79" event={"ID":"206eeab5-6be2-4392-b95d-4961a5aa1d43","Type":"ContainerDied","Data":"0a3330d68cfbd8bcf1c6d3bb13f505ece789fe4b0e0ec28ae9122a7de254101b"} Nov 03 14:04:55.275643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.275599 4924 scope.go:122] "RemoveContainer" containerID="ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5" Nov 03 14:04:55.285636 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.285617 4924 scope.go:122] "RemoveContainer" containerID="ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5" Nov 03 14:04:55.285887 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:04:55.285865 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5\": container with ID starting with ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5 not found: ID does not exist" containerID="ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5" Nov 03 14:04:55.285940 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.285900 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5"} err="failed to get container status \"ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5\": rpc error: code = NotFound desc = could not find container \"ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5\": container with ID starting with ff9683bedb6c12426d34e2c8c890ef86bb0e701034d87152d74172f8e3ae82f5 not found: ID does not exist" Nov 03 14:04:55.355274 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355243 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-model-cache\") pod \"206eeab5-6be2-4392-b95d-4961a5aa1d43\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " Nov 03 14:04:55.355274 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355278 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-dshm\") pod \"206eeab5-6be2-4392-b95d-4961a5aa1d43\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " Nov 03 14:04:55.355518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355315 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-tmp-dir\") pod \"206eeab5-6be2-4392-b95d-4961a5aa1d43\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " Nov 03 14:04:55.355518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355406 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-home\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-home\") pod \"206eeab5-6be2-4392-b95d-4961a5aa1d43\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " Nov 03 14:04:55.355518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355447 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/206eeab5-6be2-4392-b95d-4961a5aa1d43-tls-certs\" (UniqueName: \"kubernetes.io/secret/206eeab5-6be2-4392-b95d-4961a5aa1d43-tls-certs\") pod \"206eeab5-6be2-4392-b95d-4961a5aa1d43\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " Nov 03 14:04:55.355691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355521 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206eeab5-6be2-4392-b95d-4961a5aa1d43-kube-api-access-2wjgh\" (UniqueName: \"kubernetes.io/projected/206eeab5-6be2-4392-b95d-4961a5aa1d43-kube-api-access-2wjgh\") pod \"206eeab5-6be2-4392-b95d-4961a5aa1d43\" (UID: \"206eeab5-6be2-4392-b95d-4961a5aa1d43\") " Nov 03 14:04:55.355691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355525 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-model-cache" pod "206eeab5-6be2-4392-b95d-4961a5aa1d43" (UID: "206eeab5-6be2-4392-b95d-4961a5aa1d43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:04:55.355691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355673 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-home" pod "206eeab5-6be2-4392-b95d-4961a5aa1d43" (UID: "206eeab5-6be2-4392-b95d-4961a5aa1d43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:04:55.355843 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355721 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-tmp-dir" pod "206eeab5-6be2-4392-b95d-4961a5aa1d43" (UID: "206eeab5-6be2-4392-b95d-4961a5aa1d43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:04:55.355995 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355976 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:55.356089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.355995 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:55.356089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.356006 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:55.357640 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.357611 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-dshm" pod "206eeab5-6be2-4392-b95d-4961a5aa1d43" (UID: "206eeab5-6be2-4392-b95d-4961a5aa1d43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:04:55.357744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.357714 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206eeab5-6be2-4392-b95d-4961a5aa1d43-kube-api-access-2wjgh" pod "206eeab5-6be2-4392-b95d-4961a5aa1d43" (UID: "206eeab5-6be2-4392-b95d-4961a5aa1d43"). InnerVolumeSpecName "kube-api-access-2wjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:04:55.357783 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.357745 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/206eeab5-6be2-4392-b95d-4961a5aa1d43-tls-certs" pod "206eeab5-6be2-4392-b95d-4961a5aa1d43" (UID: "206eeab5-6be2-4392-b95d-4961a5aa1d43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:04:55.457507 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.457428 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjgh\" (UniqueName: \"kubernetes.io/projected/206eeab5-6be2-4392-b95d-4961a5aa1d43-kube-api-access-2wjgh\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:55.457507 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.457457 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206eeab5-6be2-4392-b95d-4961a5aa1d43-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:55.457507 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.457467 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206eeab5-6be2-4392-b95d-4961a5aa1d43-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:55.689988 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.689950 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79"] Nov 03 14:04:55.693063 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:55.693038 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-efbdec/auth-test-model-efbdec-kserve-55589d4897-d8n79"] Nov 03 14:04:56.952858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:56.952822 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="206eeab5-6be2-4392-b95d-4961a5aa1d43" path="/var/lib/kubelet/pods/206eeab5-6be2-4392-b95d-4961a5aa1d43/volumes" Nov 03 14:04:57.339315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:04:57.339237 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:02.864634 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:02.864591 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:08.130692 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:08.130604 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:13.257403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:13.257363 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:18.369952 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:18.369912 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:18.558343 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:05:18.558306 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:05:23.597441 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:23.597393 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:28.103810 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:28.103775 4924 scope.go:122] "RemoveContainer" containerID="61b2c530ce89c6684902771b17d2fd211e00c341b55257085e32107d0221f49e" Nov 03 14:05:28.113658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:28.113634 4924 scope.go:122] "RemoveContainer" containerID="4dc2fee044d5eab85e27dde416f944ea11e0aa53c0fa26341377f7ecb875a251" Nov 03 14:05:28.699424 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:28.699384 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:33.791085 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:33.791048 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:35.096812 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:35.096773 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:37.123761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:37.123729 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:37.283788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:37.283747 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:38.501669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:38.501628 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:38.958661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:38.958617 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:40.657802 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:40.657769 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:40.793568 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:40.793531 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:41.508826 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:41.508785 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:44.311678 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:44.311636 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:44.434109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:44.434061 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:45.714063 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:45.714023 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:45.837994 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:45.837957 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:47.313870 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:47.313833 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:48.558557 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:05:48.558524 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:05:49.825315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:49.825279 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:49.956545 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:49.956511 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:55.044721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:55.044687 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:05:59.891753 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:59.891719 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv"] Nov 03 14:05:59.892215 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:59.891932 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" podUID="a6a180bf-eeb3-48af-bbc3-26518c39de5f" containerName="maas-api" containerID="cri-o://37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb" gracePeriod=30 Nov 03 14:05:59.978398 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:05:59.978358 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:00.219377 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.219349 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:06:00.253823 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.253053 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls\") pod \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " Nov 03 14:06:00.253823 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.253220 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a180bf-eeb3-48af-bbc3-26518c39de5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a180bf-eeb3-48af-bbc3-26518c39de5f-openshift-service-ca\") pod \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " Nov 03 14:06:00.253823 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.253345 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-metrics-certs\") pod \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " Nov 03 14:06:00.253823 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.253379 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a180bf-eeb3-48af-bbc3-26518c39de5f-kube-api-access-nnnzk\" (UniqueName: \"kubernetes.io/projected/a6a180bf-eeb3-48af-bbc3-26518c39de5f-kube-api-access-nnnzk\") pod \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\" (UID: \"a6a180bf-eeb3-48af-bbc3-26518c39de5f\") " Nov 03 14:06:00.254268 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.253879 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a180bf-eeb3-48af-bbc3-26518c39de5f-openshift-service-ca" pod "a6a180bf-eeb3-48af-bbc3-26518c39de5f" (UID: "a6a180bf-eeb3-48af-bbc3-26518c39de5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:00.255896 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.255866 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls" pod "a6a180bf-eeb3-48af-bbc3-26518c39de5f" (UID: "a6a180bf-eeb3-48af-bbc3-26518c39de5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:06:00.256159 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.256132 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a180bf-eeb3-48af-bbc3-26518c39de5f-kube-api-access-nnnzk" pod "a6a180bf-eeb3-48af-bbc3-26518c39de5f" (UID: "a6a180bf-eeb3-48af-bbc3-26518c39de5f"). InnerVolumeSpecName "kube-api-access-nnnzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:00.256531 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.256496 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-metrics-certs" pod "a6a180bf-eeb3-48af-bbc3-26518c39de5f" (UID: "a6a180bf-eeb3-48af-bbc3-26518c39de5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:06:00.367776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.367700 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:00.367776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.367729 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a180bf-eeb3-48af-bbc3-26518c39de5f-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:00.367776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.367739 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a180bf-eeb3-48af-bbc3-26518c39de5f-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:00.367776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.367750 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnnzk\" (UniqueName: \"kubernetes.io/projected/a6a180bf-eeb3-48af-bbc3-26518c39de5f-kube-api-access-nnnzk\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:00.724934 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.724898 4924 generic.go:356] "Generic (PLEG): container finished" podID="a6a180bf-eeb3-48af-bbc3-26518c39de5f" containerID="37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb" exitCode=0 Nov 03 14:06:00.725143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.724942 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" Nov 03 14:06:00.725143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.724977 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" event={"ID":"a6a180bf-eeb3-48af-bbc3-26518c39de5f","Type":"ContainerDied","Data":"37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb"} Nov 03 14:06:00.725143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.725029 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv" event={"ID":"a6a180bf-eeb3-48af-bbc3-26518c39de5f","Type":"ContainerDied","Data":"706951343cc31f74f0b73e527c7a8d48c229c074e1b9b4361a3876083faf99bf"} Nov 03 14:06:00.725143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.725052 4924 scope.go:122] "RemoveContainer" containerID="37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb" Nov 03 14:06:00.738621 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.738589 4924 scope.go:122] "RemoveContainer" containerID="37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb" Nov 03 14:06:00.738923 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:06:00.738903 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb\": container with ID starting with 37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb not found: ID does not exist" containerID="37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb" Nov 03 14:06:00.738986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.738936 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb"} err="failed to get container status \"37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb\": rpc error: code = NotFound desc = could not find container \"37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb\": container with ID starting with 37901879e0856d22602274a446b46827b3b3dc34f5b339416b4b8037a81307bb not found: ID does not exist" Nov 03 14:06:00.882548 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.882495 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv"] Nov 03 14:06:00.927718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.927683 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ebe57-759546c9c9-g4zpv"] Nov 03 14:06:00.951003 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:00.950970 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a180bf-eeb3-48af-bbc3-26518c39de5f" path="/var/lib/kubelet/pods/a6a180bf-eeb3-48af-bbc3-26518c39de5f/volumes" Nov 03 14:06:01.853665 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:01.853625 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:04.496217 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:04.496160 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:04.657789 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:04.657751 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:06.138594 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:06.138562 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:06.719164 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:06.719126 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:06.903426 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:06.903386 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:07.114383 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:07.114305 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:10.367691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:10.367653 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:10.489338 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:10.489302 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:12.656542 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:12.656502 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:12.988297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:12.988261 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:13.641554 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:13.641522 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:13.765285 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:13.765256 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:17.004596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.004560 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:17.288732 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.288644 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4"] Nov 03 14:06:17.288889 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.288861 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" podUID="d26885da-e7a5-41f7-a869-47930fa5f1a0" containerName="maas-api" containerID="cri-o://dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c" gracePeriod=30 Nov 03 14:06:17.382109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.382056 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:17.601371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.601339 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:06:17.698498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.697548 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls\") pod \"d26885da-e7a5-41f7-a869-47930fa5f1a0\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " Nov 03 14:06:17.698498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.697632 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26885da-e7a5-41f7-a869-47930fa5f1a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26885da-e7a5-41f7-a869-47930fa5f1a0-openshift-service-ca\") pod \"d26885da-e7a5-41f7-a869-47930fa5f1a0\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " Nov 03 14:06:17.698498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.697672 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-metrics-certs\") pod \"d26885da-e7a5-41f7-a869-47930fa5f1a0\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " Nov 03 14:06:17.698498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.697787 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26885da-e7a5-41f7-a869-47930fa5f1a0-kube-api-access-ztl2l\" (UniqueName: \"kubernetes.io/projected/d26885da-e7a5-41f7-a869-47930fa5f1a0-kube-api-access-ztl2l\") pod \"d26885da-e7a5-41f7-a869-47930fa5f1a0\" (UID: \"d26885da-e7a5-41f7-a869-47930fa5f1a0\") " Nov 03 14:06:17.698498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.698052 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26885da-e7a5-41f7-a869-47930fa5f1a0-openshift-service-ca" pod "d26885da-e7a5-41f7-a869-47930fa5f1a0" (UID: "d26885da-e7a5-41f7-a869-47930fa5f1a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:17.698498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.698407 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26885da-e7a5-41f7-a869-47930fa5f1a0-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:17.701480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.701450 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-metrics-certs" pod "d26885da-e7a5-41f7-a869-47930fa5f1a0" (UID: "d26885da-e7a5-41f7-a869-47930fa5f1a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:06:17.702280 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.702255 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26885da-e7a5-41f7-a869-47930fa5f1a0-kube-api-access-ztl2l" pod "d26885da-e7a5-41f7-a869-47930fa5f1a0" (UID: "d26885da-e7a5-41f7-a869-47930fa5f1a0"). InnerVolumeSpecName "kube-api-access-ztl2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:17.707514 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.707488 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls" pod "d26885da-e7a5-41f7-a869-47930fa5f1a0" (UID: "d26885da-e7a5-41f7-a869-47930fa5f1a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:06:17.800099 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.800048 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:17.800275 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.800124 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26885da-e7a5-41f7-a869-47930fa5f1a0-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:17.800275 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.800136 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztl2l\" (UniqueName: \"kubernetes.io/projected/d26885da-e7a5-41f7-a869-47930fa5f1a0-kube-api-access-ztl2l\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:17.840221 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.840131 4924 generic.go:356] "Generic (PLEG): container finished" podID="d26885da-e7a5-41f7-a869-47930fa5f1a0" containerID="dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c" exitCode=0 Nov 03 14:06:17.840221 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.840175 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" Nov 03 14:06:17.840472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.840211 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" event={"ID":"d26885da-e7a5-41f7-a869-47930fa5f1a0","Type":"ContainerDied","Data":"dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c"} Nov 03 14:06:17.840472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.840277 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4" event={"ID":"d26885da-e7a5-41f7-a869-47930fa5f1a0","Type":"ContainerDied","Data":"16d1c117b79dbe279e016a3719aa9b3d68085dce0b500b911aea973ee708e8ff"} Nov 03 14:06:17.840472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.840307 4924 scope.go:122] "RemoveContainer" containerID="dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c" Nov 03 14:06:17.851931 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.851911 4924 scope.go:122] "RemoveContainer" containerID="dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c" Nov 03 14:06:17.852249 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:06:17.852227 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c\": container with ID starting with dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c not found: ID does not exist" containerID="dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c" Nov 03 14:06:17.852331 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:17.852262 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c"} err="failed to get container status \"dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c\": rpc error: code = NotFound desc = could not find container \"dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c\": container with ID starting with dca3adb4de3c2933f2c1fa3a12fedb812bef4096ab2e37dd92ebaa4dd578807c not found: ID does not exist" Nov 03 14:06:18.011094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:18.006899 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4"] Nov 03 14:06:18.011094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:18.009519 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-58f896-7bd7747cc5-lvzp4"] Nov 03 14:06:18.560756 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:06:18.560709 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:06:18.718550 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:18.718509 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:18.953652 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:18.953605 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26885da-e7a5-41f7-a869-47930fa5f1a0" path="/var/lib/kubelet/pods/d26885da-e7a5-41f7-a869-47930fa5f1a0/volumes" Nov 03 14:06:19.132609 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:19.132576 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:19.274985 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:19.274887 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:19.647627 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:19.647497 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:20.596887 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:20.596849 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:22.784246 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:22.784196 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:22.915959 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:22.915919 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:24.245300 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.245265 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc"] Nov 03 14:06:24.246007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.245984 4924 cpu_manager.go:414] "containerMap: removing container" podUID="206eeab5-6be2-4392-b95d-4961a5aa1d43" containerName="main" Nov 03 14:06:24.246152 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.246008 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206eeab5-6be2-4392-b95d-4961a5aa1d43" containerName="main" Nov 03 14:06:24.246152 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.246021 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d26885da-e7a5-41f7-a869-47930fa5f1a0" containerName="maas-api" Nov 03 14:06:24.246152 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.246029 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26885da-e7a5-41f7-a869-47930fa5f1a0" containerName="maas-api" Nov 03 14:06:24.246152 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.246043 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a6a180bf-eeb3-48af-bbc3-26518c39de5f" containerName="maas-api" Nov 03 14:06:24.246152 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.246052 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a180bf-eeb3-48af-bbc3-26518c39de5f" containerName="maas-api" Nov 03 14:06:24.246422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.246272 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="206eeab5-6be2-4392-b95d-4961a5aa1d43" containerName="main" Nov 03 14:06:24.246422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.246286 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26885da-e7a5-41f7-a869-47930fa5f1a0" containerName="maas-api" Nov 03 14:06:24.246422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.246298 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a180bf-eeb3-48af-bbc3-26518c39de5f" containerName="maas-api" Nov 03 14:06:24.251208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.251183 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.281503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.281466 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca78690-82de-445b-805b-05a5af62e824-openshift-service-ca\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.281646 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.281599 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhshd\" (UniqueName: \"kubernetes.io/projected/2ca78690-82de-445b-805b-05a5af62e824-kube-api-access-hhshd\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.281712 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.281698 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-metrics-certs\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.281761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.281747 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.306639 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.306605 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc"] Nov 03 14:06:24.345284 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.345248 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-aff1d2-serving-cert\"" Nov 03 14:06:24.383548 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.383510 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.383725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.383613 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca78690-82de-445b-805b-05a5af62e824-openshift-service-ca\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.383725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.383692 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhshd\" (UniqueName: \"kubernetes.io/projected/2ca78690-82de-445b-805b-05a5af62e824-kube-api-access-hhshd\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.383858 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:06:24.383768 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-serving-cert: secret "maas-api-e2e-autores-aff1d2-serving-cert" not found Nov 03 14:06:24.383858 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:06:24.383855 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls podName:2ca78690-82de-445b-805b-05a5af62e824 nodeName:}" failed. No retries permitted until 2026-11-03 14:06:24.883832828 +0000 UTC m=+3718.078236400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls") pod "maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" (UID: "2ca78690-82de-445b-805b-05a5af62e824") : secret "maas-api-e2e-autores-aff1d2-serving-cert" not found Nov 03 14:06:24.384515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.383769 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-metrics-certs\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.384836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.384813 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca78690-82de-445b-805b-05a5af62e824-openshift-service-ca\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.386932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.386909 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-metrics-certs\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.427495 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.427460 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:24.438715 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.438684 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhshd\" (UniqueName: \"kubernetes.io/projected/2ca78690-82de-445b-805b-05a5af62e824-kube-api-access-hhshd\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.588537 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.588452 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:24.704036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.704006 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:24.889515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.889426 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:24.892057 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:24.892024 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls\") pod \"maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:25.166351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:25.166319 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:25.399390 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:25.399357 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:25.527614 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:25.527465 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:25.537888 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:25.537855 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc"] Nov 03 14:06:25.538399 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:06:25.538372 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca78690_82de_445b_805b_05a5af62e824.slice/crio-51e702cb25546417025f149141b6531dd58463cb0e3f68a65724f1d9e292c0d4 WatchSource:0}: Error finding container 51e702cb25546417025f149141b6531dd58463cb0e3f68a65724f1d9e292c0d4: Status 404 returned error can't find the container with id 51e702cb25546417025f149141b6531dd58463cb0e3f68a65724f1d9e292c0d4 Nov 03 14:06:25.899424 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:25.899335 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" event={"ID":"2ca78690-82de-445b-805b-05a5af62e824","Type":"ContainerStarted","Data":"51e702cb25546417025f149141b6531dd58463cb0e3f68a65724f1d9e292c0d4"} Nov 03 14:06:26.907144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:26.907111 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" event={"ID":"2ca78690-82de-445b-805b-05a5af62e824","Type":"ContainerStarted","Data":"20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8"} Nov 03 14:06:26.907569 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:26.907237 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:27.019220 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:27.019165 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" podStartSLOduration=1.9509737280000001 podStartE2EDuration="3.019150516s" totalImagesPullingTime="1.068176788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:06:27.016145602 +0000 UTC m=+3720.210549196" watchObservedRunningTime="2026-11-03 14:06:27.019150516 +0000 UTC m=+3720.213554109" Nov 03 14:06:29.770351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:29.770312 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:32.237985 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:32.237949 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:32.350007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:32.349967 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:32.919854 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:32.919824 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:06:34.851759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:34.851717 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:37.452121 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:37.452015 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:40.165038 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:40.165003 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:45.235014 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:45.234969 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:48.564714 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:06:48.564677 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:06:50.371403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:50.371370 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:06:51.052386 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:51.052351 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf"] Nov 03 14:06:51.100260 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:51.100223 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f992c0e-ch4qf"] Nov 03 14:06:52.959009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:52.958971 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abd6683-d58f-4211-ad1f-cbc91b261196" path="/var/lib/kubelet/pods/8abd6683-d58f-4211-ad1f-cbc91b261196/volumes" Nov 03 14:06:55.503829 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:06:55.503792 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:00.237291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:00.237245 4924 patch_prober.go:29] interesting pod/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:07:00.237815 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:00.237334 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:07:00.678627 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:00.678581 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:02.050426 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.050383 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8"] Nov 03 14:07:02.093774 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.093737 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69969f78-nw5l8"] Nov 03 14:07:02.250945 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.250906 4924 generic.go:356] "Generic (PLEG): container finished" podID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerID="c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050" exitCode=0 Nov 03 14:07:02.251182 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.250987 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" event={"ID":"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd","Type":"ContainerDied","Data":"c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050"} Nov 03 14:07:02.251510 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.251492 4924 scope.go:122] "RemoveContainer" containerID="c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050" Nov 03 14:07:02.253066 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.253036 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:07:02.253175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.253119 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:07:02.253175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.253152 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:07:02.964210 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:02.964177 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fdfdc7-e018-42f1-8a2d-03f87df6f493" path="/var/lib/kubelet/pods/93fdfdc7-e018-42f1-8a2d-03f87df6f493/volumes" Nov 03 14:07:03.262058 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:03.261967 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" event={"ID":"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd","Type":"ContainerStarted","Data":"ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4"} Nov 03 14:07:03.582230 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:03.582134 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:07:03.582423 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:03.582398 4924 patch_prober.go:29] interesting pod/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 14:07:03.582477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:03.582447 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 14:07:04.022478 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.022432 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk"] Nov 03 14:07:04.025675 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.025652 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.075157 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.075126 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wd2hr\"" Nov 03 14:07:04.081986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.081954 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a63cdbb-c946-4c40-955d-05168264ef47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.082259 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.082217 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.082355 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.082315 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.082424 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.082398 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.082494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.082427 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.082494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.082459 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a63cdbb-c946-4c40-955d-05168264ef47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.082606 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.082519 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.082606 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.082567 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.086327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.082654 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7wn\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-kube-api-access-tl7wn\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.124985 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.124935 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk"] Nov 03 14:07:04.183950 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.183909 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7wn\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-kube-api-access-tl7wn\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184199 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.183954 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a63cdbb-c946-4c40-955d-05168264ef47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184199 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184018 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184199 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184133 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184209 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184237 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184271 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a63cdbb-c946-4c40-955d-05168264ef47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184319 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184367 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184648 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184601 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184757 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184743 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184841 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.184949 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.184924 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.185102 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.185080 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a63cdbb-c946-4c40-955d-05168264ef47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.186480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.186452 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.186609 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.186594 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a63cdbb-c946-4c40-955d-05168264ef47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.251761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.251726 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.253355 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.253328 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7wn\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-kube-api-access-tl7wn\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lgnwk\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.360909 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.360822 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:04.583014 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.582979 4924 patch_prober.go:29] interesting pod/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 14:07:04.583223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.583032 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 14:07:04.783997 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.783962 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk"] Nov 03 14:07:04.784309 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:07:04.784286 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a63cdbb_c946_4c40_955d_05168264ef47.slice/crio-b9351d3818009950cbc4cf682ce28be0e9bed9f728e7d724fb06d2d9fa610270 WatchSource:0}: Error finding container b9351d3818009950cbc4cf682ce28be0e9bed9f728e7d724fb06d2d9fa610270: Status 404 returned error can't find the container with id b9351d3818009950cbc4cf682ce28be0e9bed9f728e7d724fb06d2d9fa610270 Nov 03 14:07:04.786958 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.786923 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:07:04.787063 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.786990 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:07:04.787063 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:04.787018 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:07:05.292243 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.292193 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" event={"ID":"5a63cdbb-c946-4c40-955d-05168264ef47","Type":"ContainerStarted","Data":"2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed"} Nov 03 14:07:05.292477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.292294 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" event={"ID":"5a63cdbb-c946-4c40-955d-05168264ef47","Type":"ContainerStarted","Data":"b9351d3818009950cbc4cf682ce28be0e9bed9f728e7d724fb06d2d9fa610270"} Nov 03 14:07:05.361090 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.361033 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:05.363227 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.363190 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 14:07:05.363332 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.363246 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 14:07:05.404329 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.404276 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podStartSLOduration=2.404262992 podStartE2EDuration="2.404262992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:05.402064102 +0000 UTC m=+3758.596467697" watchObservedRunningTime="2026-11-03 14:07:05.404262992 +0000 UTC m=+3758.598666586" Nov 03 14:07:05.583220 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.583132 4924 patch_prober.go:29] interesting pod/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 14:07:05.583220 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.583187 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 14:07:05.724138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:05.724102 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:06.375680 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:06.375634 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 14:07:06.376346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:06.376310 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 14:07:06.582535 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:06.582499 4924 patch_prober.go:29] interesting pod/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 14:07:06.582741 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:06.582572 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 14:07:07.369105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:07.368500 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 14:07:07.369105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:07.368577 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 14:07:07.589093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:07.588552 4924 patch_prober.go:29] interesting pod/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 14:07:07.589093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:07.588668 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 14:07:08.361251 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:08.361215 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 14:07:08.361472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:08.361269 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 14:07:08.586961 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:08.586929 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:07:08.587283 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:08.587263 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:07:08.587900 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:08.587880 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:07:09.362200 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:09.362161 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 14:07:09.362611 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:09.362213 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 14:07:10.363212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:10.363160 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 14:07:10.363637 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:10.363231 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 14:07:11.258047 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:11.258000 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:11.361703 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:11.361658 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 14:07:11.361919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:11.361746 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 14:07:12.361836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:12.361792 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 14:07:12.362256 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:12.361871 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 14:07:13.628704 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:13.628652 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:13.630113 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:13.630085 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:13.630714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:13.630693 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:07:15.569907 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:15.569873 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:15.697042 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:15.697006 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:16.897646 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:16.897614 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:18.139951 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:18.139909 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:18.262118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:18.262038 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:18.622191 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:07:18.622153 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:07:21.684433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:21.684374 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:23.555705 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:23.555636 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:23.688011 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:23.687976 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:23.925327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:23.925289 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:24.300866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:24.300779 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:24.425902 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:24.425867 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:24.714270 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:24.714238 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:28.895258 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:28.895223 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:30.638538 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:30.638496 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:30.775498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:30.775462 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:30.984246 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:30.984202 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:31.110183 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:31.110144 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:31.230702 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:31.230660 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:31.389161 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:31.389042 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:31.521759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:31.521721 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:34.473233 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:34.473195 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:36.853713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:36.853675 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:37.191952 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:37.191882 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:37.335485 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:37.335444 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:37.837427 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:37.837393 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:39.355358 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:39.355324 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:39.475208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:39.475171 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:40.357360 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:40.357325 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:45.921102 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:45.921041 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:46.060208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:46.060172 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:48.559469 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:07:48.559427 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:07:49.581501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:49.581468 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:50.381951 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.381912 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s"] Nov 03 14:07:50.382290 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.382259 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" podUID="f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" containerName="maas-api" containerID="cri-o://63747fb8e4659b16cd78075f634a49cf418dd6935c7f6fa9d6d26e506acd2c90" gracePeriod=30 Nov 03 14:07:50.476119 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.476064 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:50.607882 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.607685 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:50.688191 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.688152 4924 generic.go:356] "Generic (PLEG): container finished" podID="f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" containerID="63747fb8e4659b16cd78075f634a49cf418dd6935c7f6fa9d6d26e506acd2c90" exitCode=0 Nov 03 14:07:50.688399 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.688233 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" event={"ID":"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862","Type":"ContainerDied","Data":"63747fb8e4659b16cd78075f634a49cf418dd6935c7f6fa9d6d26e506acd2c90"} Nov 03 14:07:50.731833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.731806 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:07:50.787093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.785930 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-kube-api-access-7m9vd\" (UniqueName: \"kubernetes.io/projected/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-kube-api-access-7m9vd\") pod \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " Nov 03 14:07:50.787093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.786004 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-openshift-service-ca\") pod \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " Nov 03 14:07:50.787093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.786144 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-metrics-certs\") pod \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " Nov 03 14:07:50.787093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.786213 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls\") pod \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\" (UID: \"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862\") " Nov 03 14:07:50.817223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.792292 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-kube-api-access-7m9vd" pod "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" (UID: "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862"). InnerVolumeSpecName "kube-api-access-7m9vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:50.817223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.797785 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-openshift-service-ca" pod "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" (UID: "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:50.817223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.802106 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls" pod "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" (UID: "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:50.817223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.804211 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-metrics-certs" pod "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" (UID: "f627e6ce-9bc2-4c69-abb4-8fa3df2a9862"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:50.888786 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.888749 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9vd\" (UniqueName: \"kubernetes.io/projected/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-kube-api-access-7m9vd\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:50.888786 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.888780 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:50.888786 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.888792 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:50.889112 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:50.888800 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:51.064114 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:51.064026 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:51.253704 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:51.253587 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:51.696621 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:51.696590 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" Nov 03 14:07:51.697109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:51.696589 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s" event={"ID":"f627e6ce-9bc2-4c69-abb4-8fa3df2a9862","Type":"ContainerDied","Data":"96b1fd3c8ac813ecb00885b633661250308e049a1d24c521845eb3fab99e7a46"} Nov 03 14:07:51.697109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:51.696754 4924 scope.go:122] "RemoveContainer" containerID="63747fb8e4659b16cd78075f634a49cf418dd6935c7f6fa9d6d26e506acd2c90" Nov 03 14:07:51.851888 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:51.851845 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s"] Nov 03 14:07:51.898066 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:51.898023 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c4e398-7b5d8b6885-9vv5s"] Nov 03 14:07:52.212765 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:52.212734 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:52.340456 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:52.340415 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:52.739196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:52.739162 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:52.753330 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:52.753297 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g"] Nov 03 14:07:52.753536 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:52.753509 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" containerID="cri-o://826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca" gracePeriod=30 Nov 03 14:07:52.951499 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:52.951468 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" path="/var/lib/kubelet/pods/f627e6ce-9bc2-4c69-abb4-8fa3df2a9862/volumes" Nov 03 14:07:53.540726 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:53.540689 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:53.682539 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:53.682494 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:53.821598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:53.821510 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:54.896155 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:54.896108 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:55.128248 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:55.128205 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:55.250732 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:55.250694 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:55.327595 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:55.327559 4924 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:07:55.327751 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:55.327635 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:07:56.412640 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:56.412593 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:56.951690 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:56.951655 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc"] Nov 03 14:07:56.951912 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:56.951882 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" podUID="2ca78690-82de-445b-805b-05a5af62e824" containerName="maas-api" containerID="cri-o://20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8" gracePeriod=30 Nov 03 14:07:57.039959 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.039920 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:57.254320 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.254287 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:07:57.385359 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.385327 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca78690-82de-445b-805b-05a5af62e824-kube-api-access-hhshd\" (UniqueName: \"kubernetes.io/projected/2ca78690-82de-445b-805b-05a5af62e824-kube-api-access-hhshd\") pod \"2ca78690-82de-445b-805b-05a5af62e824\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " Nov 03 14:07:57.385560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.385454 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls\") pod \"2ca78690-82de-445b-805b-05a5af62e824\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " Nov 03 14:07:57.385560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.385491 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-metrics-certs\") pod \"2ca78690-82de-445b-805b-05a5af62e824\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " Nov 03 14:07:57.385560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.385527 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca78690-82de-445b-805b-05a5af62e824-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca78690-82de-445b-805b-05a5af62e824-openshift-service-ca\") pod \"2ca78690-82de-445b-805b-05a5af62e824\" (UID: \"2ca78690-82de-445b-805b-05a5af62e824\") " Nov 03 14:07:57.386274 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.386244 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca78690-82de-445b-805b-05a5af62e824-openshift-service-ca" pod "2ca78690-82de-445b-805b-05a5af62e824" (UID: "2ca78690-82de-445b-805b-05a5af62e824"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:57.388017 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.387995 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca78690-82de-445b-805b-05a5af62e824-kube-api-access-hhshd" pod "2ca78690-82de-445b-805b-05a5af62e824" (UID: "2ca78690-82de-445b-805b-05a5af62e824"). InnerVolumeSpecName "kube-api-access-hhshd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:57.388176 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.388160 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-metrics-certs" pod "2ca78690-82de-445b-805b-05a5af62e824" (UID: "2ca78690-82de-445b-805b-05a5af62e824"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:57.390188 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.390162 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls" pod "2ca78690-82de-445b-805b-05a5af62e824" (UID: "2ca78690-82de-445b-805b-05a5af62e824"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:57.487301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.487218 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:57.487301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.487258 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca78690-82de-445b-805b-05a5af62e824-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:57.487301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.487274 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca78690-82de-445b-805b-05a5af62e824-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:57.487301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.487288 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhshd\" (UniqueName: \"kubernetes.io/projected/2ca78690-82de-445b-805b-05a5af62e824-kube-api-access-hhshd\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:57.582590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.582528 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:57.698455 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.698418 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:57.749166 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.749034 4924 generic.go:356] "Generic (PLEG): container finished" podID="2ca78690-82de-445b-805b-05a5af62e824" containerID="20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8" exitCode=0 Nov 03 14:07:57.749166 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.749110 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" Nov 03 14:07:57.749413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.749297 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" event={"ID":"2ca78690-82de-445b-805b-05a5af62e824","Type":"ContainerDied","Data":"20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8"} Nov 03 14:07:57.749413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.749353 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc" event={"ID":"2ca78690-82de-445b-805b-05a5af62e824","Type":"ContainerDied","Data":"51e702cb25546417025f149141b6531dd58463cb0e3f68a65724f1d9e292c0d4"} Nov 03 14:07:57.749413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.749386 4924 scope.go:122] "RemoveContainer" containerID="20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8" Nov 03 14:07:57.774212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.774181 4924 scope.go:122] "RemoveContainer" containerID="20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8" Nov 03 14:07:57.774742 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:07:57.774647 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8\": container with ID starting with 20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8 not found: ID does not exist" containerID="20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8" Nov 03 14:07:57.774742 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.774690 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8"} err="failed to get container status \"20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8\": rpc error: code = NotFound desc = could not find container \"20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8\": container with ID starting with 20fba3b7560e0b0546fd4e4da1b6a8b34cdd41cebc3aff744505c627153174a8 not found: ID does not exist" Nov 03 14:07:57.906930 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.906900 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc"] Nov 03 14:07:57.909811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:57.909784 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-aff1d2-7b57c655ff-rmkqc"] Nov 03 14:07:58.087887 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.087857 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:07:58.201372 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201326 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-token\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-token\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201372 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201385 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-certs\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201436 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-data\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201505 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-podinfo\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201556 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-kube-api-access-p7ncz\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-kube-api-access-p7ncz\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201593 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-socket\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201627 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-envoy\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201964 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201675 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1520ae8-820c-4f50-b823-770d7fa776f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1520ae8-820c-4f50-b823-770d7fa776f4-istiod-ca-cert\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201964 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201797 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-credential-socket\") pod \"a1520ae8-820c-4f50-b823-770d7fa776f4\" (UID: \"a1520ae8-820c-4f50-b823-770d7fa776f4\") " Nov 03 14:07:58.201964 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.201808 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-certs" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.202274 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.202249 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.202886 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.202568 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-data" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.202886 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.202581 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-credential-socket" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.202886 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.202850 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-socket" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.202886 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.202869 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1520ae8-820c-4f50-b823-770d7fa776f4-istiod-ca-cert" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:58.204180 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.204155 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-token" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:58.204297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.204230 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-kube-api-access-p7ncz" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "kube-api-access-p7ncz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:58.205372 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.205349 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-envoy" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.206328 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.206304 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-podinfo" pod "a1520ae8-820c-4f50-b823-770d7fa776f4" (UID: "a1520ae8-820c-4f50-b823-770d7fa776f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:07:58.303214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.303178 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.303214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.303209 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.303214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.303218 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.303487 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.303228 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.303487 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.303239 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7ncz\" (UniqueName: \"kubernetes.io/projected/a1520ae8-820c-4f50-b823-770d7fa776f4-kube-api-access-p7ncz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.303487 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.303253 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.303487 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.303265 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1520ae8-820c-4f50-b823-770d7fa776f4-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.303487 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.303272 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1520ae8-820c-4f50-b823-770d7fa776f4-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.757351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.757317 4924 generic.go:356] "Generic (PLEG): container finished" podID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerID="826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca" exitCode=0 Nov 03 14:07:58.757806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.757370 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" Nov 03 14:07:58.757806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.757419 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" event={"ID":"a1520ae8-820c-4f50-b823-770d7fa776f4","Type":"ContainerDied","Data":"826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca"} Nov 03 14:07:58.757806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.757521 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g" event={"ID":"a1520ae8-820c-4f50-b823-770d7fa776f4","Type":"ContainerDied","Data":"c6af66dae23bab4a1d277ba0e886206ac37248724e0640ddf797aa5a214f0558"} Nov 03 14:07:58.757806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.757553 4924 scope.go:122] "RemoveContainer" containerID="826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca" Nov 03 14:07:58.768986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.768970 4924 scope.go:122] "RemoveContainer" containerID="826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca" Nov 03 14:07:58.769247 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:07:58.769227 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca\": container with ID starting with 826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca not found: ID does not exist" containerID="826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca" Nov 03 14:07:58.769311 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.769256 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca"} err="failed to get container status \"826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca\": rpc error: code = NotFound desc = could not find container \"826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca\": container with ID starting with 826e23336737fb7157b38e0a4e07c98c4991d1f19c0d5881f47e80b2e74e13ca not found: ID does not exist" Nov 03 14:07:58.914341 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.914307 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g"] Nov 03 14:07:58.951602 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.951569 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca78690-82de-445b-805b-05a5af62e824" path="/var/lib/kubelet/pods/2ca78690-82de-445b-805b-05a5af62e824/volumes" Nov 03 14:07:58.961025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:58.960997 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-tgp5g"] Nov 03 14:07:59.062140 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:59.062028 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:59.691246 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:59.691207 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:07:59.810213 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:07:59.810182 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:00.997086 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:00.997025 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" path="/var/lib/kubelet/pods/a1520ae8-820c-4f50-b823-770d7fa776f4/volumes" Nov 03 14:08:02.040470 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:02.040430 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:03.168403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:03.168365 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:03.345158 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:03.345119 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:04.504860 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.504824 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:04.547503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.547465 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76"] Nov 03 14:08:04.547988 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.547975 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2ca78690-82de-445b-805b-05a5af62e824" containerName="maas-api" Nov 03 14:08:04.548033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.547990 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca78690-82de-445b-805b-05a5af62e824" containerName="maas-api" Nov 03 14:08:04.548033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.548016 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" containerName="maas-api" Nov 03 14:08:04.548033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.548022 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" containerName="maas-api" Nov 03 14:08:04.548033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.548031 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" Nov 03 14:08:04.548207 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.548036 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" Nov 03 14:08:04.548207 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.548160 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca78690-82de-445b-805b-05a5af62e824" containerName="maas-api" Nov 03 14:08:04.548207 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.548173 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f627e6ce-9bc2-4c69-abb4-8fa3df2a9862" containerName="maas-api" Nov 03 14:08:04.548207 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.548181 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1520ae8-820c-4f50-b823-770d7fa776f4" containerName="istio-proxy" Nov 03 14:08:04.551859 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.551691 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.652391 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.652353 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76"] Nov 03 14:08:04.679706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.679668 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.679894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.679727 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5w9\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-kube-api-access-7g5w9\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.679894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.679810 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46fe371-f392-41df-bde9-847cbccef8a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.679894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.679880 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46fe371-f392-41df-bde9-847cbccef8a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.680027 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.679931 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.680027 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.679972 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.680130 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.680031 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.680130 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.680054 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.680233 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.680193 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.780986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.780903 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.780986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.780943 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.780986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.780990 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781036 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781108 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5w9\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-kube-api-access-7g5w9\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781142 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46fe371-f392-41df-bde9-847cbccef8a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781182 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46fe371-f392-41df-bde9-847cbccef8a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781237 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781284 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781655 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781349 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781655 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781423 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.781894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.781870 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.782055 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.782034 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46fe371-f392-41df-bde9-847cbccef8a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.782247 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.782224 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.783844 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.783807 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.784339 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.784320 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46fe371-f392-41df-bde9-847cbccef8a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.835138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.835104 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.835138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.835125 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5w9\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-kube-api-access-7g5w9\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-kws76\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:04.874834 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:04.874787 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:05.297882 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.297851 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76"] Nov 03 14:08:05.300105 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:08:05.300061 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46fe371_f392_41df_bde9_847cbccef8a6.slice/crio-e8cb42f3ac334340f38b674453e7a517780d239983554b106168ace82c0216bd WatchSource:0}: Error finding container e8cb42f3ac334340f38b674453e7a517780d239983554b106168ace82c0216bd: Status 404 returned error can't find the container with id e8cb42f3ac334340f38b674453e7a517780d239983554b106168ace82c0216bd Nov 03 14:08:05.302701 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.302657 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:05.302787 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.302727 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:05.302787 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.302760 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:05.707024 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.706992 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:05.842149 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.842107 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" event={"ID":"d46fe371-f392-41df-bde9-847cbccef8a6","Type":"ContainerStarted","Data":"f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960"} Nov 03 14:08:05.842346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.842173 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" event={"ID":"d46fe371-f392-41df-bde9-847cbccef8a6","Type":"ContainerStarted","Data":"e8cb42f3ac334340f38b674453e7a517780d239983554b106168ace82c0216bd"} Nov 03 14:08:05.874901 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.874862 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:05.959269 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:05.959135 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podStartSLOduration=1.959114661 podStartE2EDuration="1.959114661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:05.953749668 +0000 UTC m=+3819.148153267" watchObservedRunningTime="2026-11-03 14:08:05.959114661 +0000 UTC m=+3819.153518261" Nov 03 14:08:06.876833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:06.876744 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-kws76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 14:08:06.876833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:06.876801 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 14:08:07.718920 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:07.718882 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:07.853495 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:07.853451 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:07.875084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:07.875047 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-kws76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 14:08:07.875245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:07.875121 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 14:08:08.875154 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:08.875123 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-kws76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 14:08:08.875570 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:08.875175 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 14:08:09.432177 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:09.432113 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:09.763609 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:09.763479 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:09.885055 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:09.885014 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-kws76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 14:08:09.885957 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:09.885677 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 14:08:09.893118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:09.893086 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5"] Nov 03 14:08:09.893366 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:09.893334 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" containerID="cri-o://ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4" gracePeriod=30 Nov 03 14:08:09.904990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:09.904965 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:10.875928 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:10.875889 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-kws76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 14:08:10.876297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:10.875954 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 14:08:11.747490 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:11.747449 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:11.875277 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:11.875234 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-kws76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 14:08:11.875482 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:11.875296 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 14:08:12.366094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:12.363615 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:12.613849 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:12.613807 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:12.875250 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:12.875215 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-kws76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 14:08:12.875757 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:12.875725 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 14:08:14.144766 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:14.144728 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:14.145538 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:14.145518 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:14.146393 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:14.146367 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:08:15.132920 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.132883 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:15.166528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.166452 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:08:15.272204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.272163 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istiod-ca-cert\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.272392 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.272275 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-token\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-token\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.272392 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.272323 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-kube-api-access-xkwrs\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-kube-api-access-xkwrs\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.272530 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.272420 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-credential-socket\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.272761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.272631 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istiod-ca-cert" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:08:15.272761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.272753 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-credential-socket" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:08:15.273106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.273028 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-data\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.273710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.273382 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-envoy\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.273710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.273433 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-podinfo\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.273710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.273472 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-certs\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.273710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.273512 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-socket\") pod \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\" (UID: \"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd\") " Nov 03 14:08:15.273963 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.273749 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-data" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:08:15.274024 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.273974 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-certs" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:08:15.274673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.274213 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-socket" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:08:15.274673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.274530 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.274673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.274553 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.274673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.274568 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.274673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.274581 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.274673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.274595 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.275340 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.275309 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-token" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:08:15.276095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.276042 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-kube-api-access-xkwrs" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "kube-api-access-xkwrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:08:15.277235 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.277204 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-podinfo" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:08:15.277616 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.277558 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-envoy" pod "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" (UID: "3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:08:15.375833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.375797 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.375833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.375833 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.376055 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.375849 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.376055 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.375863 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkwrs\" (UniqueName: \"kubernetes.io/projected/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd-kube-api-access-xkwrs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:15.958337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.958305 4924 generic.go:356] "Generic (PLEG): container finished" podID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerID="ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4" exitCode=0 Nov 03 14:08:15.958531 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.958377 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" Nov 03 14:08:15.958531 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.958398 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" event={"ID":"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd","Type":"ContainerDied","Data":"ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4"} Nov 03 14:08:15.958531 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.958455 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5" event={"ID":"3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd","Type":"ContainerDied","Data":"05f931d4c1e1dc86741367fcf3472411313017a258092eb7e1ec53379ba29535"} Nov 03 14:08:15.958531 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.958487 4924 scope.go:122] "RemoveContainer" containerID="ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4" Nov 03 14:08:15.970791 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.970519 4924 scope.go:122] "RemoveContainer" containerID="c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050" Nov 03 14:08:15.981332 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.981305 4924 scope.go:122] "RemoveContainer" containerID="ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4" Nov 03 14:08:15.981605 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:08:15.981587 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4\": container with ID starting with ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4 not found: ID does not exist" containerID="ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4" Nov 03 14:08:15.981653 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.981617 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4"} err="failed to get container status \"ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4\": rpc error: code = NotFound desc = could not find container \"ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4\": container with ID starting with ba98becedf814acb2c0144781da4639e384c437ae5a383afc9a808516027ffb4 not found: ID does not exist" Nov 03 14:08:15.981653 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.981636 4924 scope.go:122] "RemoveContainer" containerID="c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050" Nov 03 14:08:15.981875 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:08:15.981860 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050\": container with ID starting with c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050 not found: ID does not exist" containerID="c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050" Nov 03 14:08:15.981914 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:15.981879 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050"} err="failed to get container status \"c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050\": rpc error: code = NotFound desc = could not find container \"c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050\": container with ID starting with c4acffef9d7efd597fca6d3908391cce1574e59748ddc0c3901527649360d050 not found: ID does not exist" Nov 03 14:08:16.024180 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:16.024142 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:16.168478 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:16.168443 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5"] Nov 03 14:08:16.215443 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:16.215336 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:16.215647 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:16.215628 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-aff1d2-openshift-default-847f8b466c-xxzq5"] Nov 03 14:08:16.380630 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:16.380584 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:16.954981 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:16.954950 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" path="/var/lib/kubelet/pods/3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd/volumes" Nov 03 14:08:18.560753 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:08:18.560713 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:08:19.647658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:19.647586 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:19.875370 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:19.875333 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:20.792595 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:20.792556 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:25.972896 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:25.972859 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:28.248803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:28.248769 4924 scope.go:122] "RemoveContainer" containerID="c775f9b6a26e3320190f9bdcceb682bb6b6b912fe0f14a64f16f2a92f0f6de53" Nov 03 14:08:28.260538 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:28.260513 4924 scope.go:122] "RemoveContainer" containerID="33a5593016daa1f7ad3a67142005f8d5cc4ca01bf552d4aa1285a662597b40a4" Nov 03 14:08:31.091039 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:31.091004 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:36.185643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:36.185604 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:38.154653 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:38.154599 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:38.312733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:38.312694 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:41.264846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:41.264810 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:46.387624 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:46.387592 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:48.560324 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:08:48.560282 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:08:51.588025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:51.587983 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:55.506993 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.506951 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk"] Nov 03 14:08:55.507529 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.507515 4924 cpu_manager.go:414] "containerMap: removing container" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" Nov 03 14:08:55.507582 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.507530 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" Nov 03 14:08:55.507637 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.507627 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" Nov 03 14:08:55.511138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.511121 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.518897 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.518866 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.519030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.519011 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.519161 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.519141 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6cx\" (UniqueName: \"kubernetes.io/projected/3a808efd-c00c-46ba-84bf-5e90c875c256-kube-api-access-fg6cx\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.567612 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.567578 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk"] Nov 03 14:08:55.609006 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.608978 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:08:55.609234 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.609216 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:08:55.620436 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.620410 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6cx\" (UniqueName: \"kubernetes.io/projected/3a808efd-c00c-46ba-84bf-5e90c875c256-kube-api-access-fg6cx\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.620550 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.620497 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.620615 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.620536 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.621184 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.621161 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.623212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.623191 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.675389 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.675363 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6cx\" (UniqueName: \"kubernetes.io/projected/3a808efd-c00c-46ba-84bf-5e90c875c256-kube-api-access-fg6cx\") pod \"maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:55.825097 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:55.824979 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:56.193023 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:56.192995 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk"] Nov 03 14:08:56.194461 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:08:56.194435 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a808efd_c00c_46ba_84bf_5e90c875c256.slice/crio-bb74d4130749fa7c48f53e22588ac419953d9b5b9c554611d7fdbfcef6a38443 WatchSource:0}: Error finding container bb74d4130749fa7c48f53e22588ac419953d9b5b9c554611d7fdbfcef6a38443: Status 404 returned error can't find the container with id bb74d4130749fa7c48f53e22588ac419953d9b5b9c554611d7fdbfcef6a38443 Nov 03 14:08:56.271188 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:56.271145 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" event={"ID":"3a808efd-c00c-46ba-84bf-5e90c875c256","Type":"ContainerStarted","Data":"bb74d4130749fa7c48f53e22588ac419953d9b5b9c554611d7fdbfcef6a38443"} Nov 03 14:08:56.908840 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:56.908805 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:08:57.279024 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:57.278940 4924 generic.go:356] "Generic (PLEG): container finished" podID="3a808efd-c00c-46ba-84bf-5e90c875c256" containerID="efe450de8bada22c9a3a50de81e5baa654ff0ba4125e916c98c0587ee6dfb0e2" exitCode=0 Nov 03 14:08:57.279200 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:57.279034 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" event={"ID":"3a808efd-c00c-46ba-84bf-5e90c875c256","Type":"ContainerDied","Data":"efe450de8bada22c9a3a50de81e5baa654ff0ba4125e916c98c0587ee6dfb0e2"} Nov 03 14:08:58.433828 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.433796 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:58.454854 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.454825 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-openshift-ca\") pod \"3a808efd-c00c-46ba-84bf-5e90c875c256\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " Nov 03 14:08:58.454989 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.454923 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-xks-ca\") pod \"3a808efd-c00c-46ba-84bf-5e90c875c256\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " Nov 03 14:08:58.455043 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.454995 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a808efd-c00c-46ba-84bf-5e90c875c256-kube-api-access-fg6cx\" (UniqueName: \"kubernetes.io/projected/3a808efd-c00c-46ba-84bf-5e90c875c256-kube-api-access-fg6cx\") pod \"3a808efd-c00c-46ba-84bf-5e90c875c256\" (UID: \"3a808efd-c00c-46ba-84bf-5e90c875c256\") " Nov 03 14:08:58.455217 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.455194 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-openshift-ca" pod "3a808efd-c00c-46ba-84bf-5e90c875c256" (UID: "3a808efd-c00c-46ba-84bf-5e90c875c256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:08:58.455341 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.455326 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:58.457061 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.457039 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a808efd-c00c-46ba-84bf-5e90c875c256-kube-api-access-fg6cx" pod "3a808efd-c00c-46ba-84bf-5e90c875c256" (UID: "3a808efd-c00c-46ba-84bf-5e90c875c256"). InnerVolumeSpecName "kube-api-access-fg6cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:08:58.457186 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.457167 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-xks-ca" pod "3a808efd-c00c-46ba-84bf-5e90c875c256" (UID: "3a808efd-c00c-46ba-84bf-5e90c875c256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:08:58.556015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.555934 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a808efd-c00c-46ba-84bf-5e90c875c256-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:58.556015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:58.555966 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg6cx\" (UniqueName: \"kubernetes.io/projected/3a808efd-c00c-46ba-84bf-5e90c875c256-kube-api-access-fg6cx\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:59.292391 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:59.292302 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" Nov 03 14:08:59.292391 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:59.292330 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk" event={"ID":"3a808efd-c00c-46ba-84bf-5e90c875c256","Type":"ContainerDied","Data":"bb74d4130749fa7c48f53e22588ac419953d9b5b9c554611d7fdbfcef6a38443"} Nov 03 14:08:59.292391 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:08:59.292379 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb74d4130749fa7c48f53e22588ac419953d9b5b9c554611d7fdbfcef6a38443" Nov 03 14:09:01.965010 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:01.964977 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:03.168515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:03.168476 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:03.315179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:03.315136 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:05.024365 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.024329 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48"] Nov 03 14:09:05.024912 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.024894 4924 cpu_manager.go:414] "containerMap: removing container" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" Nov 03 14:09:05.024958 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.024914 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" Nov 03 14:09:05.024958 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.024956 4924 cpu_manager.go:414] "containerMap: removing container" podUID="3a808efd-c00c-46ba-84bf-5e90c875c256" containerName="revoke-keys" Nov 03 14:09:05.025036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.024961 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a808efd-c00c-46ba-84bf-5e90c875c256" containerName="revoke-keys" Nov 03 14:09:05.025036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.025033 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a808efd-c00c-46ba-84bf-5e90c875c256" containerName="revoke-keys" Nov 03 14:09:05.025140 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.025048 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3615ac1f-0dc3-46d3-b97a-c7e062b5c9dd" containerName="istio-proxy" Nov 03 14:09:05.028462 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.028440 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.072543 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.072505 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8"] Nov 03 14:09:05.077128 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.077101 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.126756 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.126714 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48"] Nov 03 14:09:05.132696 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.132660 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.132866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.132746 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be04d1a7-2306-4e9b-90c4-354e2521e60f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.132866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.132786 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.132866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.132802 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9713196-25d4-4c50-9327-2d4c1010baf6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.133048 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.132898 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.133048 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.132926 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.133048 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.132943 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj4b\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-kube-api-access-blj4b\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.133048 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.132974 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.133297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133059 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.133297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133094 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psssr\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-kube-api-access-psssr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.133297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133112 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.133297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133242 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.133297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133282 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.133572 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133355 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.133572 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133392 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.133572 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133410 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.133572 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133432 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.133572 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.133455 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.175671 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.175633 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8"] Nov 03 14:09:05.234544 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234505 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be04d1a7-2306-4e9b-90c4-354e2521e60f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.234719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234560 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.234719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234576 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9713196-25d4-4c50-9327-2d4c1010baf6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.234719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234609 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.234719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234630 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.234719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234650 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj4b\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-kube-api-access-blj4b\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.234719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234688 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234730 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234772 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psssr\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-kube-api-access-psssr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.235059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.234797 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235123 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.235438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235267 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235332 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.235438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235346 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235368 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235423 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235455 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235463 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9713196-25d4-4c50-9327-2d4c1010baf6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235482 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235530 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235561 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235578 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235613 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235631 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.235767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235730 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be04d1a7-2306-4e9b-90c4-354e2521e60f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.236294 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235770 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.236294 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.235953 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.236294 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.236008 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.237522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.237501 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.237703 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.237680 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.238051 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.238030 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.238150 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.238033 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.289899 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.289817 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.290135 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.289952 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psssr\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-kube-api-access-psssr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-wtf48\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.290305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.290282 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj4b\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-kube-api-access-blj4b\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.290386 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.290296 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-82cq8\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.343085 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.343027 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:05.391888 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.391856 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:05.771703 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.771673 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48"] Nov 03 14:09:05.773570 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:09:05.773540 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe04d1a7_2306_4e9b_90c4_354e2521e60f.slice/crio-37412ffeba26c3eaff17567f620554f20413435c47ddc93ccffa868205025241 WatchSource:0}: Error finding container 37412ffeba26c3eaff17567f620554f20413435c47ddc93ccffa868205025241: Status 404 returned error can't find the container with id 37412ffeba26c3eaff17567f620554f20413435c47ddc93ccffa868205025241 Nov 03 14:09:05.776577 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.776544 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:05.776652 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.776610 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:05.776652 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.776645 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:05.822204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.822177 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8"] Nov 03 14:09:05.822840 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:09:05.822814 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9713196_25d4_4c50_9327_2d4c1010baf6.slice/crio-cd2981f2e242939a3a1e025276c71e56255846a148a5730105bd40398a3e7ce6 WatchSource:0}: Error finding container cd2981f2e242939a3a1e025276c71e56255846a148a5730105bd40398a3e7ce6: Status 404 returned error can't find the container with id cd2981f2e242939a3a1e025276c71e56255846a148a5730105bd40398a3e7ce6 Nov 03 14:09:05.825717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.825682 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:05.825796 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.825748 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:05.825796 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:05.825779 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:06.345853 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.345807 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" event={"ID":"f9713196-25d4-4c50-9327-2d4c1010baf6","Type":"ContainerStarted","Data":"1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626"} Nov 03 14:09:06.346416 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.345881 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" event={"ID":"f9713196-25d4-4c50-9327-2d4c1010baf6","Type":"ContainerStarted","Data":"cd2981f2e242939a3a1e025276c71e56255846a148a5730105bd40398a3e7ce6"} Nov 03 14:09:06.349433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.349404 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" event={"ID":"be04d1a7-2306-4e9b-90c4-354e2521e60f","Type":"ContainerStarted","Data":"69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7"} Nov 03 14:09:06.349635 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.349465 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" event={"ID":"be04d1a7-2306-4e9b-90c4-354e2521e60f","Type":"ContainerStarted","Data":"37412ffeba26c3eaff17567f620554f20413435c47ddc93ccffa868205025241"} Nov 03 14:09:06.392943 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.392902 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:06.394509 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.394471 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:09:06.394664 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.394537 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 14:09:06.474818 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.474757 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podStartSLOduration=1.4747391570000001 podStartE2EDuration="1.474739157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:06.464516137 +0000 UTC m=+3879.658919732" watchObservedRunningTime="2026-11-03 14:09:06.474739157 +0000 UTC m=+3879.669142751" Nov 03 14:09:06.590175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:06.590109 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podStartSLOduration=2.5900889940000003 podStartE2EDuration="2.590088994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:06.585785101 +0000 UTC m=+3879.780188697" watchObservedRunningTime="2026-11-03 14:09:06.590088994 +0000 UTC m=+3879.784492583" Nov 03 14:09:07.180696 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:07.180569 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:07.346559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:07.346468 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:07.353567 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:07.353209 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-wtf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 14:09:07.353567 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:07.353275 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 14:09:07.396223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:07.393572 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:09:07.396223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:07.393645 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 14:09:08.345271 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:08.345233 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-wtf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 14:09:08.345558 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:08.345530 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 14:09:08.392722 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:08.392691 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:09:08.393224 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:08.393198 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 14:09:09.344095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:09.343604 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-wtf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 14:09:09.344095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:09.343669 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 14:09:09.392460 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:09.392413 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:09:09.392679 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:09.392483 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 14:09:10.344489 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:10.344452 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-wtf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 14:09:10.345091 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:10.345037 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 14:09:10.396093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:10.395635 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:09:10.396093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:10.395695 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 14:09:11.345208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:11.345166 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-wtf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 14:09:11.345822 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:11.345789 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 14:09:11.393044 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:11.392717 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:09:11.393044 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:11.392777 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 14:09:12.286431 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:12.286394 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:12.344632 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:12.344573 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-wtf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 14:09:12.344842 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:12.344661 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 14:09:12.392951 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:12.392895 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:09:12.393675 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:12.393641 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 14:09:13.343792 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:13.343740 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-wtf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 14:09:13.344021 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:13.343809 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 14:09:13.393209 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:13.393163 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:09:13.393738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:13.393252 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 14:09:14.827057 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:14.825754 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:14.827057 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:14.826981 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:14.832477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:14.832413 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:09:14.871990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:14.871957 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:14.872367 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:14.872312 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:14.873871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:14.873843 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:09:17.078362 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.078324 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw"] Nov 03 14:09:17.082947 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.082921 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.133744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.133706 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d6963e-serving-cert\"" Nov 03 14:09:17.138321 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.138288 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw"] Nov 03 14:09:17.173621 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.173583 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/783e0122-fab1-404c-a88f-c59693e538a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.173806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.173631 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6lr4\" (UniqueName: \"kubernetes.io/projected/783e0122-fab1-404c-a88f-c59693e538a9-kube-api-access-f6lr4\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.173806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.173656 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-metrics-certs\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.173806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.173792 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.259113 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.259054 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:17.275427 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.275401 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/783e0122-fab1-404c-a88f-c59693e538a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.275543 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.275436 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6lr4\" (UniqueName: \"kubernetes.io/projected/783e0122-fab1-404c-a88f-c59693e538a9-kube-api-access-f6lr4\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.275543 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.275457 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-metrics-certs\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.275543 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.275500 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.275724 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:09:17.275703 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-serving-cert: secret "maas-api-e2e-inf-a-d6963e-serving-cert" not found Nov 03 14:09:17.275784 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:09:17.275774 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls podName:783e0122-fab1-404c-a88f-c59693e538a9 nodeName:}" failed. No retries permitted until 2026-11-03 14:09:17.775759457 +0000 UTC m=+3890.970163042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls") pod "maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" (UID: "783e0122-fab1-404c-a88f-c59693e538a9") : secret "maas-api-e2e-inf-a-d6963e-serving-cert" not found Nov 03 14:09:17.276162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.276137 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/783e0122-fab1-404c-a88f-c59693e538a9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.277969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.277943 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-metrics-certs\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.344418 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.344335 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6lr4\" (UniqueName: \"kubernetes.io/projected/783e0122-fab1-404c-a88f-c59693e538a9-kube-api-access-f6lr4\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.421659 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.421626 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:17.547020 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.546976 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:17.781100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.781035 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:17.783564 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:17.783529 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:18.001180 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:18.001143 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:18.380652 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:18.380617 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw"] Nov 03 14:09:18.381229 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:09:18.381196 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783e0122_fab1_404c_a88f_c59693e538a9.slice/crio-a9ba2f1fc2649252374f8c9af4fd843bb5da540112df2859cf377905ddfcad78 WatchSource:0}: Error finding container a9ba2f1fc2649252374f8c9af4fd843bb5da540112df2859cf377905ddfcad78: Status 404 returned error can't find the container with id a9ba2f1fc2649252374f8c9af4fd843bb5da540112df2859cf377905ddfcad78 Nov 03 14:09:18.382884 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:18.382863 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:09:18.484728 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:18.484690 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" event={"ID":"783e0122-fab1-404c-a88f-c59693e538a9","Type":"ContainerStarted","Data":"a9ba2f1fc2649252374f8c9af4fd843bb5da540112df2859cf377905ddfcad78"} Nov 03 14:09:18.559314 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:09:18.559278 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:09:19.493590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:19.493492 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" event={"ID":"783e0122-fab1-404c-a88f-c59693e538a9","Type":"ContainerStarted","Data":"02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd"} Nov 03 14:09:19.493590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:19.493558 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:19.628204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:19.628149 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" podStartSLOduration=1.6997112859999999 podStartE2EDuration="2.628133964s" totalImagesPullingTime="928.422678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:19.626612391 +0000 UTC m=+3892.821015984" watchObservedRunningTime="2026-11-03 14:09:19.628133964 +0000 UTC m=+3892.822537558" Nov 03 14:09:21.945113 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:21.945053 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:23.751246 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:23.751204 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:25.176421 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:25.176382 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:25.506804 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:25.506725 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:09:25.703577 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:25.703546 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:25.852004 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:25.851911 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:26.345271 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:26.345240 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:27.201433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:27.201398 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:27.344380 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:27.344336 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:27.608924 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:27.608845 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:09:27.619359 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:27.619334 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:09:27.703787 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:27.703749 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:29.190936 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:29.190902 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:31.715393 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:31.715353 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:31.899711 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:31.899678 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:32.044846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:32.044743 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:34.617846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:34.617814 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:34.783578 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:34.783545 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:34.910528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:34.910495 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:38.214165 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:38.214056 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:38.409174 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:38.409136 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:38.532027 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:38.531934 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:40.545760 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:40.545718 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:45.786878 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:45.786838 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:48.560254 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:09:48.560209 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:09:50.928310 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:50.928275 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:55.087926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:55.087890 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:55.210850 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:55.210811 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:55.758284 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:55.758246 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:09:56.053266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:09:56.053183 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:01.110218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:01.110180 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:06.197017 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:06.196979 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:11.395417 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:11.395358 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:16.606412 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:16.606373 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:18.558633 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:10:18.558597 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:10:20.961946 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:20.961908 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d"] Nov 03 14:10:20.962518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:20.962130 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" podUID="ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" containerName="maas-api" containerID="cri-o://b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda" gracePeriod=30 Nov 03 14:10:21.058418 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.058379 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:21.276537 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.276506 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:10:21.402089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.401692 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-kube-api-access-jp87l\" (UniqueName: \"kubernetes.io/projected/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-kube-api-access-jp87l\") pod \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " Nov 03 14:10:21.402089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.401764 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-metrics-certs\") pod \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " Nov 03 14:10:21.402089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.401792 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls\") pod \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " Nov 03 14:10:21.402089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.401851 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-openshift-service-ca\") pod \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\" (UID: \"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88\") " Nov 03 14:10:21.406092 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.402799 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-openshift-service-ca" pod "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" (UID: "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:10:21.458277 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.458224 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-metrics-certs" pod "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" (UID: "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:10:21.471699 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.471486 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls" pod "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" (UID: "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:10:21.471699 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.471650 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-kube-api-access-jp87l" pod "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" (UID: "ea0f9151-ad2a-4ac2-b667-f58b1ec98e88"). InnerVolumeSpecName "kube-api-access-jp87l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:10:21.503800 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.503706 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp87l\" (UniqueName: \"kubernetes.io/projected/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-kube-api-access-jp87l\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:21.503800 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.503748 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:21.503800 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.503762 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:21.503800 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.503775 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:21.909881 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.909844 4924 generic.go:356] "Generic (PLEG): container finished" podID="ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" containerID="b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda" exitCode=0 Nov 03 14:10:21.910098 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.909890 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" Nov 03 14:10:21.910098 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.909927 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" event={"ID":"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88","Type":"ContainerDied","Data":"b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda"} Nov 03 14:10:21.910098 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.909985 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d" event={"ID":"ea0f9151-ad2a-4ac2-b667-f58b1ec98e88","Type":"ContainerDied","Data":"711ec7b4a425c23fa84f9ec9b0dbfebc360ea6ca6c102e876817d825a76963da"} Nov 03 14:10:21.910098 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.910008 4924 scope.go:122] "RemoveContainer" containerID="b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda" Nov 03 14:10:21.930951 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.930698 4924 scope.go:122] "RemoveContainer" containerID="b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda" Nov 03 14:10:21.931718 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:10:21.931691 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda\": container with ID starting with b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda not found: ID does not exist" containerID="b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda" Nov 03 14:10:21.931845 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:21.931724 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda"} err="failed to get container status \"b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda\": rpc error: code = NotFound desc = could not find container \"b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda\": container with ID starting with b6d9c4c441c278211e52e3f04ea4fb5f75b9306f7dbb7b788d7d28810fa68bda not found: ID does not exist" Nov 03 14:10:22.084946 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:22.084915 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d"] Nov 03 14:10:22.130021 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:22.129986 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d5d415-789d459c57-77j5d"] Nov 03 14:10:22.841145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:22.841059 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:22.952656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:22.952615 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" path="/var/lib/kubelet/pods/ea0f9151-ad2a-4ac2-b667-f58b1ec98e88/volumes" Nov 03 14:10:22.970674 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:22.970638 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:23.097885 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:23.097799 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk"] Nov 03 14:10:23.146344 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:23.146310 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d5d415-8crqk"] Nov 03 14:10:23.321477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:23.321438 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h"] Nov 03 14:10:23.321765 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:23.321717 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" containerID="cri-o://da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3" gracePeriod=30 Nov 03 14:10:23.340167 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:23.340123 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:24.209788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.209753 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:24.731191 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.731157 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j"] Nov 03 14:10:24.731803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.731781 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" containerName="maas-api" Nov 03 14:10:24.731920 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.731805 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" containerName="maas-api" Nov 03 14:10:24.731981 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.731946 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0f9151-ad2a-4ac2-b667-f58b1ec98e88" containerName="maas-api" Nov 03 14:10:24.736791 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.736769 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.798189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.798151 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b6c8c7-openshift-default-dockercfg-lfqfr\"" Nov 03 14:10:24.808993 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.808821 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.919669 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istiod-ca-cert\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.919955 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-podinfo\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.919989 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-envoy\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.920167 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-socket\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.920215 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-token\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.920349 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-credential-socket\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.920487 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-certs\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.920627 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qf9\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-kube-api-access-z7qf9\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.920827 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-data\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:24.931795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.923651 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j"] Nov 03 14:10:24.973874 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.973842 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a808efd-c00c-46ba-84bf-5e90c875c256" path="/var/lib/kubelet/pods/3a808efd-c00c-46ba-84bf-5e90c875c256/volumes" Nov 03 14:10:24.975235 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:24.975213 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:25.021991 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.021896 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-data\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.021991 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.021988 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istiod-ca-cert\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022021 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-podinfo\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022041 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-envoy\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022064 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-socket\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022110 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-token\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022140 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-credential-socket\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022175 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-certs\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022207 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qf9\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-kube-api-access-z7qf9\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022662 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022439 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-data\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022753 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022733 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-socket\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.022924 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.022904 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-credential-socket\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.023173 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.023152 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-certs\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.023379 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.023358 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istiod-ca-cert\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.025598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.025575 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-envoy\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.062573 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.058756 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-podinfo\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.077245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.077216 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qf9\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-kube-api-access-z7qf9\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.080515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.080486 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-token\") pod \"e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.351544 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.351439 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:25.790539 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:10:25.790501 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081b76f0_1d17_41ed_b0ba_9256d5bd919c.slice/crio-4e8bccb76d3bb94397a9cf2e46d6895a1968209a10f398bcece95cd5b7eb202f WatchSource:0}: Error finding container 4e8bccb76d3bb94397a9cf2e46d6895a1968209a10f398bcece95cd5b7eb202f: Status 404 returned error can't find the container with id 4e8bccb76d3bb94397a9cf2e46d6895a1968209a10f398bcece95cd5b7eb202f Nov 03 14:10:25.790744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.790715 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j"] Nov 03 14:10:25.793262 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.793225 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:10:25.793356 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.793292 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:10:25.793356 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.793322 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:10:25.979654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.979613 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" event={"ID":"081b76f0-1d17-41ed-b0ba-9256d5bd919c","Type":"ContainerStarted","Data":"6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139"} Nov 03 14:10:25.979871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:25.979680 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" event={"ID":"081b76f0-1d17-41ed-b0ba-9256d5bd919c","Type":"ContainerStarted","Data":"4e8bccb76d3bb94397a9cf2e46d6895a1968209a10f398bcece95cd5b7eb202f"} Nov 03 14:10:26.134724 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:26.134624 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" podStartSLOduration=2.134609222 podStartE2EDuration="2.134609222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:26.131990179 +0000 UTC m=+3959.326393783" watchObservedRunningTime="2026-11-03 14:10:26.134609222 +0000 UTC m=+3959.329012815" Nov 03 14:10:26.352679 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:26.352640 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:26.358486 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:26.358450 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:26.992327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:26.992292 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:26.993259 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:26.993232 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:10:28.124375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.124330 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:28.253851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.253817 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:28.577358 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.577335 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:10:28.669267 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669232 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-data\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.669529 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669283 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-socket\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.669529 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669461 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-podinfo\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.669529 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669508 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-credential-socket\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.669713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669530 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-data" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:10:28.669713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669543 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-kube-api-access-f2497\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-kube-api-access-f2497\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.669713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669583 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-socket" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:10:28.669713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669617 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d8e5c4-5ba1-481e-9a0b-cec934989777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d8e5c4-5ba1-481e-9a0b-cec934989777-istiod-ca-cert\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.669713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669674 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-certs\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.669713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669710 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-token\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-token\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.670036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669781 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-envoy\") pod \"21d8e5c4-5ba1-481e-9a0b-cec934989777\" (UID: \"21d8e5c4-5ba1-481e-9a0b-cec934989777\") " Nov 03 14:10:28.670036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669841 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-credential-socket" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:10:28.670036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669946 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d8e5c4-5ba1-481e-9a0b-cec934989777-istiod-ca-cert" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:10:28.670036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.669944 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-certs" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:10:28.670347 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.670326 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.670415 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.670353 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d8e5c4-5ba1-481e-9a0b-cec934989777-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.670415 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.670369 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.670415 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.670384 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.670415 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.670399 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.671837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.671807 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-kube-api-access-f2497" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "kube-api-access-f2497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:10:28.671837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.671809 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-podinfo" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:10:28.672021 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.671888 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-token" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:10:28.672021 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.671976 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-envoy" pod "21d8e5c4-5ba1-481e-9a0b-cec934989777" (UID: "21d8e5c4-5ba1-481e-9a0b-cec934989777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:10:28.739439 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.739353 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:28.771772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.771732 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.771772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.771766 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.771772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.771777 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d8e5c4-5ba1-481e-9a0b-cec934989777-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.772121 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.771786 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2497\" (UniqueName: \"kubernetes.io/projected/21d8e5c4-5ba1-481e-9a0b-cec934989777-kube-api-access-f2497\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:10:28.874484 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:28.874429 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:29.010375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.010290 4924 generic.go:356] "Generic (PLEG): container finished" podID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerID="da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3" exitCode=0 Nov 03 14:10:29.010375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.010344 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" Nov 03 14:10:29.010596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.010372 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" event={"ID":"21d8e5c4-5ba1-481e-9a0b-cec934989777","Type":"ContainerDied","Data":"da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3"} Nov 03 14:10:29.010596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.010434 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h" event={"ID":"21d8e5c4-5ba1-481e-9a0b-cec934989777","Type":"ContainerDied","Data":"144a1a2dfd4d0148f3e579761d7189a3827c64a01559109a7493c609558f8248"} Nov 03 14:10:29.010596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.010458 4924 scope.go:122] "RemoveContainer" containerID="da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3" Nov 03 14:10:29.021137 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.021116 4924 scope.go:122] "RemoveContainer" containerID="607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682" Nov 03 14:10:29.032482 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.032462 4924 scope.go:122] "RemoveContainer" containerID="da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3" Nov 03 14:10:29.032831 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:10:29.032801 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3\": container with ID starting with da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3 not found: ID does not exist" containerID="da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3" Nov 03 14:10:29.032957 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.032833 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3"} err="failed to get container status \"da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3\": rpc error: code = NotFound desc = could not find container \"da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3\": container with ID starting with da2d24640a7c940eb13570064a2065d088f2fe61d5470919487075ceadb6cae3 not found: ID does not exist" Nov 03 14:10:29.032957 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.032856 4924 scope.go:122] "RemoveContainer" containerID="607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682" Nov 03 14:10:29.033183 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:10:29.033162 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682\": container with ID starting with 607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682 not found: ID does not exist" containerID="607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682" Nov 03 14:10:29.033266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.033189 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682"} err="failed to get container status \"607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682\": rpc error: code = NotFound desc = could not find container \"607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682\": container with ID starting with 607fb059a0121e796842de7439d14be87082dee93e22d6487308af521feb7682 not found: ID does not exist" Nov 03 14:10:29.086758 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.086722 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:29.168433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.168393 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h"] Nov 03 14:10:29.214882 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.214841 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d5d415-openshift-default-6899d86897-pjk5h"] Nov 03 14:10:29.232721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:29.232682 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:30.951724 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:30.951693 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" path="/var/lib/kubelet/pods/21d8e5c4-5ba1-481e-9a0b-cec934989777/volumes" Nov 03 14:10:32.166835 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:32.166792 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:32.343106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:32.341533 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:34.416163 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:34.416126 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:38.796493 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:38.796446 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:40.007357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:40.007300 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:45.440246 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:45.440211 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:48.558728 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:10:48.558690 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:10:50.571448 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:50.571419 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:53.873680 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:53.873642 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:56.066635 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:56.066568 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:10:59.656865 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:10:59.656830 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:00.794576 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.794535 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:00.945003 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.944950 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:00.953579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.953552 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz"] Nov 03 14:11:00.954042 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.954030 4924 cpu_manager.go:414] "containerMap: removing container" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" Nov 03 14:11:00.954120 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.954044 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" Nov 03 14:11:00.954120 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.954064 4924 cpu_manager.go:414] "containerMap: removing container" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" Nov 03 14:11:00.954120 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.954100 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" Nov 03 14:11:00.954232 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.954200 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" Nov 03 14:11:00.958589 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:00.958559 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.014969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.014898 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6375e7-serving-cert\"" Nov 03 14:11:01.062304 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.062202 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.062304 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.062269 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87kgd\" (UniqueName: \"kubernetes.io/projected/724623c4-14a4-41c5-93b8-9be7c80dcd32-kube-api-access-87kgd\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.062524 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.062396 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/724623c4-14a4-41c5-93b8-9be7c80dcd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.062574 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.062553 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-metrics-certs\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.067628 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.067597 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz"] Nov 03 14:11:01.159742 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.159707 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:01.164049 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.164018 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-metrics-certs\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.164203 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.164189 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.164263 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.164233 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87kgd\" (UniqueName: \"kubernetes.io/projected/724623c4-14a4-41c5-93b8-9be7c80dcd32-kube-api-access-87kgd\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.164327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.164261 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/724623c4-14a4-41c5-93b8-9be7c80dcd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.164390 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:11:01.164355 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-serving-cert: secret "maas-api-e2e-inf-a-6375e7-serving-cert" not found Nov 03 14:11:01.164446 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:11:01.164434 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls podName:724623c4-14a4-41c5-93b8-9be7c80dcd32 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:01.664411018 +0000 UTC m=+3994.858814597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls") pod "maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" (UID: "724623c4-14a4-41c5-93b8-9be7c80dcd32") : secret "maas-api-e2e-inf-a-6375e7-serving-cert" not found Nov 03 14:11:01.164949 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.164929 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/724623c4-14a4-41c5-93b8-9be7c80dcd32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.166700 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.166677 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-metrics-certs\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.227042 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.220359 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87kgd\" (UniqueName: \"kubernetes.io/projected/724623c4-14a4-41c5-93b8-9be7c80dcd32-kube-api-access-87kgd\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.398223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.398141 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:01.544299 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.544259 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:01.671637 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.671601 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.674171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.674147 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls\") pod \"maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:01.891096 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:01.881088 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:02.305511 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:02.305480 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz"] Nov 03 14:11:02.306634 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:11:02.306603 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724623c4_14a4_41c5_93b8_9be7c80dcd32.slice/crio-6465b6c07e897ae05563b177744ef8b5443af2f1aa3ed856871d6f6015007a07 WatchSource:0}: Error finding container 6465b6c07e897ae05563b177744ef8b5443af2f1aa3ed856871d6f6015007a07: Status 404 returned error can't find the container with id 6465b6c07e897ae05563b177744ef8b5443af2f1aa3ed856871d6f6015007a07 Nov 03 14:11:03.252275 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:03.252236 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" event={"ID":"724623c4-14a4-41c5-93b8-9be7c80dcd32","Type":"ContainerStarted","Data":"6465b6c07e897ae05563b177744ef8b5443af2f1aa3ed856871d6f6015007a07"} Nov 03 14:11:04.261605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:04.261567 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" event={"ID":"724623c4-14a4-41c5-93b8-9be7c80dcd32","Type":"ContainerStarted","Data":"6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2"} Nov 03 14:11:04.262140 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:04.261787 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:04.373089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:04.373027 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" podStartSLOduration=3.432588339 podStartE2EDuration="4.373012373s" totalImagesPullingTime="940.424034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:04.369900753 +0000 UTC m=+3997.564304347" watchObservedRunningTime="2026-11-03 14:11:04.373012373 +0000 UTC m=+3997.567415966" Nov 03 14:11:06.026639 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:06.026604 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:07.074155 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:07.074047 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:10.315564 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:10.315529 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:11:12.722559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:12.722523 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:15.106210 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:15.106178 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:18.154198 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:18.154160 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:18.560224 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:11:18.560177 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:11:21.521897 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:21.521852 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:22.397045 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:22.397011 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:22.516938 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:22.516859 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:23.543631 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:23.543584 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:27.138805 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:27.138745 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:29.219712 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:29.219673 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:34.508040 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:34.507980 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:39.147744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:39.147708 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:39.315735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:39.315694 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:39.597911 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:39.597875 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:44.686115 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:44.686064 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:48.560139 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:11:48.560104 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:11:49.876494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:49.876456 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:54.981099 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:54.981056 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:11:59.997382 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:11:59.997352 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:00.338536 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:00.333828 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:00.462512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:00.462475 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:05.364506 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:05.364455 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:10.460188 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:10.460144 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:15.538571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:15.538531 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:18.562859 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:12:18.562816 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:12:19.866345 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:19.866306 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw"] Nov 03 14:12:19.866911 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:19.866526 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" podUID="783e0122-fab1-404c-a88f-c59693e538a9" containerName="maas-api" containerID="cri-o://02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd" gracePeriod=30 Nov 03 14:12:19.962880 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:19.962844 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:20.243162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.243126 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:12:20.352371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.352327 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/783e0122-fab1-404c-a88f-c59693e538a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/783e0122-fab1-404c-a88f-c59693e538a9-openshift-service-ca\") pod \"783e0122-fab1-404c-a88f-c59693e538a9\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " Nov 03 14:12:20.352592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.352384 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783e0122-fab1-404c-a88f-c59693e538a9-kube-api-access-f6lr4\" (UniqueName: \"kubernetes.io/projected/783e0122-fab1-404c-a88f-c59693e538a9-kube-api-access-f6lr4\") pod \"783e0122-fab1-404c-a88f-c59693e538a9\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " Nov 03 14:12:20.352592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.352462 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-metrics-certs\") pod \"783e0122-fab1-404c-a88f-c59693e538a9\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " Nov 03 14:12:20.352592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.352560 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls\") pod \"783e0122-fab1-404c-a88f-c59693e538a9\" (UID: \"783e0122-fab1-404c-a88f-c59693e538a9\") " Nov 03 14:12:20.353771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.353742 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783e0122-fab1-404c-a88f-c59693e538a9-openshift-service-ca" pod "783e0122-fab1-404c-a88f-c59693e538a9" (UID: "783e0122-fab1-404c-a88f-c59693e538a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:20.357469 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.357423 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-metrics-certs" pod "783e0122-fab1-404c-a88f-c59693e538a9" (UID: "783e0122-fab1-404c-a88f-c59693e538a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:20.358594 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.358515 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls" pod "783e0122-fab1-404c-a88f-c59693e538a9" (UID: "783e0122-fab1-404c-a88f-c59693e538a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:20.358594 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.358559 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783e0122-fab1-404c-a88f-c59693e538a9-kube-api-access-f6lr4" pod "783e0122-fab1-404c-a88f-c59693e538a9" (UID: "783e0122-fab1-404c-a88f-c59693e538a9"). InnerVolumeSpecName "kube-api-access-f6lr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:20.454515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.454417 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:20.454515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.454459 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/783e0122-fab1-404c-a88f-c59693e538a9-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:20.454515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.454483 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6lr4\" (UniqueName: \"kubernetes.io/projected/783e0122-fab1-404c-a88f-c59693e538a9-kube-api-access-f6lr4\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:20.454515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.454500 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e0122-fab1-404c-a88f-c59693e538a9-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:20.828731 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.828644 4924 generic.go:356] "Generic (PLEG): container finished" podID="783e0122-fab1-404c-a88f-c59693e538a9" containerID="02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd" exitCode=0 Nov 03 14:12:20.828731 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.828710 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" Nov 03 14:12:20.828935 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.828739 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" event={"ID":"783e0122-fab1-404c-a88f-c59693e538a9","Type":"ContainerDied","Data":"02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd"} Nov 03 14:12:20.828935 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.828809 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw" event={"ID":"783e0122-fab1-404c-a88f-c59693e538a9","Type":"ContainerDied","Data":"a9ba2f1fc2649252374f8c9af4fd843bb5da540112df2859cf377905ddfcad78"} Nov 03 14:12:20.828935 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.828840 4924 scope.go:122] "RemoveContainer" containerID="02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd" Nov 03 14:12:20.841159 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.841141 4924 scope.go:122] "RemoveContainer" containerID="02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd" Nov 03 14:12:20.841460 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:12:20.841440 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd\": container with ID starting with 02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd not found: ID does not exist" containerID="02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd" Nov 03 14:12:20.841532 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.841475 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd"} err="failed to get container status \"02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd\": rpc error: code = NotFound desc = could not find container \"02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd\": container with ID starting with 02d0eff0d7470ec778093437525cdbbf647eab563d52bd4bda1b18fb769237fd not found: ID does not exist" Nov 03 14:12:20.993245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:20.993199 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw"] Nov 03 14:12:21.039952 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:21.039918 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6963e-65fbdbc8c6-97cqw"] Nov 03 14:12:21.791623 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:21.791585 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:22.239372 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:22.239335 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:22.371580 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:22.371528 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:22.951237 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:22.951205 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783e0122-fab1-404c-a88f-c59693e538a9" path="/var/lib/kubelet/pods/783e0122-fab1-404c-a88f-c59693e538a9/volumes" Nov 03 14:12:23.809560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:23.809523 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:24.390959 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:24.390511 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:27.011673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:27.011630 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:27.172434 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:27.172397 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:27.336151 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:27.326346 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:27.468354 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:27.468318 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:30.667783 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:30.667722 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:32.941887 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:32.941852 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:34.939771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:34.939734 4924 generic.go:356] "Generic (PLEG): container finished" podID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerID="6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139" exitCode=0 Nov 03 14:12:34.940308 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:34.939802 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" event={"ID":"081b76f0-1d17-41ed-b0ba-9256d5bd919c","Type":"ContainerDied","Data":"6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139"} Nov 03 14:12:34.940308 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:34.940221 4924 scope.go:122] "RemoveContainer" containerID="6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139" Nov 03 14:12:34.941775 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:34.941743 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:12:34.941841 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:34.941813 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:12:34.941884 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:34.941847 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:12:35.964762 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:35.964726 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" event={"ID":"081b76f0-1d17-41ed-b0ba-9256d5bd919c","Type":"ContainerStarted","Data":"0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761"} Nov 03 14:12:36.357404 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:36.357298 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:12:36.357593 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:36.357553 4924 patch_prober.go:29] interesting pod/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 14:12:36.357665 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:36.357608 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 14:12:36.447870 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:36.447727 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:37.358683 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:37.358581 4924 patch_prober.go:29] interesting pod/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 14:12:37.358683 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:37.358653 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 14:12:37.746183 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:37.746145 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:37.934856 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:37.934806 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:38.353948 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:38.353481 4924 patch_prober.go:29] interesting pod/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 14:12:38.353948 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:38.353557 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 14:12:39.359522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:39.359476 4924 patch_prober.go:29] interesting pod/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 14:12:39.360196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:39.360113 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 14:12:39.500421 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:39.500372 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:39.683322 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:39.683276 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:40.352522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:40.352478 4924 patch_prober.go:29] interesting pod/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 14:12:40.352871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:40.352830 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 14:12:41.357348 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:41.357312 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:12:41.357838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:41.357798 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:12:41.358695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:41.358674 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:12:42.628999 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:42.628962 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:42.758196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:42.758151 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:42.883018 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:42.882938 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:44.372655 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:44.372624 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:44.888171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:44.888137 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:45.041094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:45.035691 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:47.064407 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:47.064367 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:47.716614 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:47.714669 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:48.077437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:48.077188 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:48.664537 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:12:48.664495 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:12:50.958426 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:50.958386 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:52.402998 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:52.402963 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:52.613496 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:52.613459 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:52.741176 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:52.741130 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:55.691464 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:55.691433 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:55.894567 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:55.892361 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:12:56.844296 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:12:56.844258 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:01.861691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:01.861656 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:07.040017 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:07.039980 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:07.168615 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:07.168578 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:12.092814 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:12.092775 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:17.163185 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:17.163139 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:18.558661 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:13:18.558619 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:13:22.194594 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:22.194543 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:25.746740 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.746699 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7"] Nov 03 14:13:25.747277 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.747262 4924 cpu_manager.go:414] "containerMap: removing container" podUID="783e0122-fab1-404c-a88f-c59693e538a9" containerName="maas-api" Nov 03 14:13:25.747277 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.747277 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783e0122-fab1-404c-a88f-c59693e538a9" containerName="maas-api" Nov 03 14:13:25.747411 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.747358 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d8e5c4-5ba1-481e-9a0b-cec934989777" containerName="istio-proxy" Nov 03 14:13:25.747411 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.747371 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="783e0122-fab1-404c-a88f-c59693e538a9" containerName="maas-api" Nov 03 14:13:25.749571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.749550 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:25.804782 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.804745 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7"] Nov 03 14:13:25.847387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.847349 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:13:25.847387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.847362 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:13:25.850581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.850552 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh228\" (UniqueName: \"kubernetes.io/projected/d9a8e841-252f-4396-94cd-32536e18f32c-kube-api-access-gh228\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:25.850726 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.850605 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:25.850825 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.850802 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:25.952425 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.952373 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:25.952630 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.952602 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:25.952708 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.952663 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh228\" (UniqueName: \"kubernetes.io/projected/d9a8e841-252f-4396-94cd-32536e18f32c-kube-api-access-gh228\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:25.953144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.953118 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:25.955110 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:25.955058 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:26.006991 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:26.006917 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh228\" (UniqueName: \"kubernetes.io/projected/d9a8e841-252f-4396-94cd-32536e18f32c-kube-api-access-gh228\") pod \"maas-api-revoke-sub-61bc07e9-z55m7\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:26.062881 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:26.062832 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:26.434605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:26.434574 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7"] Nov 03 14:13:26.437125 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:13:26.437090 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a8e841_252f_4396_94cd_32536e18f32c.slice/crio-ef6c53d77e3d9e564c31b1255d6a34e5385ac796f7fa7f271b96771dd82f32f8 WatchSource:0}: Error finding container ef6c53d77e3d9e564c31b1255d6a34e5385ac796f7fa7f271b96771dd82f32f8: Status 404 returned error can't find the container with id ef6c53d77e3d9e564c31b1255d6a34e5385ac796f7fa7f271b96771dd82f32f8 Nov 03 14:13:27.219634 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:27.219595 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:27.418678 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:27.418636 4924 generic.go:356] "Generic (PLEG): container finished" podID="d9a8e841-252f-4396-94cd-32536e18f32c" containerID="1a8b044808468dc236d3c3f79f60e4f16030715bbad33afe22843cf4438bba98" exitCode=0 Nov 03 14:13:27.418874 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:27.418682 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" event={"ID":"d9a8e841-252f-4396-94cd-32536e18f32c","Type":"ContainerDied","Data":"1a8b044808468dc236d3c3f79f60e4f16030715bbad33afe22843cf4438bba98"} Nov 03 14:13:27.418874 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:27.418816 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" event={"ID":"d9a8e841-252f-4396-94cd-32536e18f32c","Type":"ContainerStarted","Data":"ef6c53d77e3d9e564c31b1255d6a34e5385ac796f7fa7f271b96771dd82f32f8"} Nov 03 14:13:28.579215 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.579188 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:28.686173 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.686067 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-xks-ca\") pod \"d9a8e841-252f-4396-94cd-32536e18f32c\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " Nov 03 14:13:28.686358 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.686179 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a8e841-252f-4396-94cd-32536e18f32c-kube-api-access-gh228\" (UniqueName: \"kubernetes.io/projected/d9a8e841-252f-4396-94cd-32536e18f32c-kube-api-access-gh228\") pod \"d9a8e841-252f-4396-94cd-32536e18f32c\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " Nov 03 14:13:28.686358 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.686250 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-openshift-ca\") pod \"d9a8e841-252f-4396-94cd-32536e18f32c\" (UID: \"d9a8e841-252f-4396-94cd-32536e18f32c\") " Nov 03 14:13:28.686655 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.686625 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-openshift-ca" pod "d9a8e841-252f-4396-94cd-32536e18f32c" (UID: "d9a8e841-252f-4396-94cd-32536e18f32c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:13:28.688312 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.688286 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-xks-ca" pod "d9a8e841-252f-4396-94cd-32536e18f32c" (UID: "d9a8e841-252f-4396-94cd-32536e18f32c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:13:28.688725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.688695 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a8e841-252f-4396-94cd-32536e18f32c-kube-api-access-gh228" pod "d9a8e841-252f-4396-94cd-32536e18f32c" (UID: "d9a8e841-252f-4396-94cd-32536e18f32c"). InnerVolumeSpecName "kube-api-access-gh228". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:13:28.787466 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.787421 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh228\" (UniqueName: \"kubernetes.io/projected/d9a8e841-252f-4396-94cd-32536e18f32c-kube-api-access-gh228\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:28.787466 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.787458 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:28.787466 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:28.787473 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a8e841-252f-4396-94cd-32536e18f32c-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:29.432788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:29.432761 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" Nov 03 14:13:29.432993 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:29.432784 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7" event={"ID":"d9a8e841-252f-4396-94cd-32536e18f32c","Type":"ContainerDied","Data":"ef6c53d77e3d9e564c31b1255d6a34e5385ac796f7fa7f271b96771dd82f32f8"} Nov 03 14:13:29.432993 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:29.432837 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef6c53d77e3d9e564c31b1255d6a34e5385ac796f7fa7f271b96771dd82f32f8" Nov 03 14:13:32.295557 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:32.295515 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:32.773128 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:32.773032 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:36.355175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:36.355136 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:37.699022 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:37.698979 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:38.617447 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:38.617411 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:39.483484 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:39.483447 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:39.604657 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:39.604611 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:39.854409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:39.854329 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:42.064203 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:42.064164 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:43.249863 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:43.249831 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:44.085488 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:44.085448 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:48.365503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:48.365462 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:48.559444 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:13:48.559404 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:13:53.369723 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:53.369685 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:55.234837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:55.234785 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:55.393768 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:55.393729 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:57.713579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:57.713542 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz"] Nov 03 14:13:57.714006 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:57.713744 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" podUID="724623c4-14a4-41c5-93b8-9be7c80dcd32" containerName="maas-api" containerID="cri-o://6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2" gracePeriod=30 Nov 03 14:13:57.801815 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:57.801780 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:13:58.036834 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.036800 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:13:58.094212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.094179 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls\") pod \"724623c4-14a4-41c5-93b8-9be7c80dcd32\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " Nov 03 14:13:58.094427 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.094259 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724623c4-14a4-41c5-93b8-9be7c80dcd32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/724623c4-14a4-41c5-93b8-9be7c80dcd32-openshift-service-ca\") pod \"724623c4-14a4-41c5-93b8-9be7c80dcd32\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " Nov 03 14:13:58.094427 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.094354 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724623c4-14a4-41c5-93b8-9be7c80dcd32-kube-api-access-87kgd\" (UniqueName: \"kubernetes.io/projected/724623c4-14a4-41c5-93b8-9be7c80dcd32-kube-api-access-87kgd\") pod \"724623c4-14a4-41c5-93b8-9be7c80dcd32\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " Nov 03 14:13:58.094427 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.094414 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-metrics-certs\") pod \"724623c4-14a4-41c5-93b8-9be7c80dcd32\" (UID: \"724623c4-14a4-41c5-93b8-9be7c80dcd32\") " Nov 03 14:13:58.094805 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.094775 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724623c4-14a4-41c5-93b8-9be7c80dcd32-openshift-service-ca" pod "724623c4-14a4-41c5-93b8-9be7c80dcd32" (UID: "724623c4-14a4-41c5-93b8-9be7c80dcd32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:13:58.094955 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.094868 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/724623c4-14a4-41c5-93b8-9be7c80dcd32-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:58.097024 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.097001 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724623c4-14a4-41c5-93b8-9be7c80dcd32-kube-api-access-87kgd" pod "724623c4-14a4-41c5-93b8-9be7c80dcd32" (UID: "724623c4-14a4-41c5-93b8-9be7c80dcd32"). InnerVolumeSpecName "kube-api-access-87kgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:13:58.100041 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.098731 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls" pod "724623c4-14a4-41c5-93b8-9be7c80dcd32" (UID: "724623c4-14a4-41c5-93b8-9be7c80dcd32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:13:58.100041 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.098834 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-metrics-certs" pod "724623c4-14a4-41c5-93b8-9be7c80dcd32" (UID: "724623c4-14a4-41c5-93b8-9be7c80dcd32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:13:58.200755 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.200091 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:58.200755 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.200135 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87kgd\" (UniqueName: \"kubernetes.io/projected/724623c4-14a4-41c5-93b8-9be7c80dcd32-kube-api-access-87kgd\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:58.200755 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.200151 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/724623c4-14a4-41c5-93b8-9be7c80dcd32-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:58.731094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.726101 4924 generic.go:356] "Generic (PLEG): container finished" podID="724623c4-14a4-41c5-93b8-9be7c80dcd32" containerID="6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2" exitCode=0 Nov 03 14:13:58.731094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.726350 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" event={"ID":"724623c4-14a4-41c5-93b8-9be7c80dcd32","Type":"ContainerDied","Data":"6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2"} Nov 03 14:13:58.731094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.726422 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" event={"ID":"724623c4-14a4-41c5-93b8-9be7c80dcd32","Type":"ContainerDied","Data":"6465b6c07e897ae05563b177744ef8b5443af2f1aa3ed856871d6f6015007a07"} Nov 03 14:13:58.731094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.726482 4924 scope.go:122] "RemoveContainer" containerID="6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2" Nov 03 14:13:58.731094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.726683 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz" Nov 03 14:13:58.777146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.771888 4924 scope.go:122] "RemoveContainer" containerID="6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2" Nov 03 14:13:58.777146 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:13:58.772560 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2\": container with ID starting with 6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2 not found: ID does not exist" containerID="6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2" Nov 03 14:13:58.777146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.772600 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2"} err="failed to get container status \"6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2\": rpc error: code = NotFound desc = could not find container \"6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2\": container with ID starting with 6bb144cba3a7ae9a369487ac48b5946fff2d8b06ef93ddc969114577461961c2 not found: ID does not exist" Nov 03 14:13:58.903507 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.903472 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz"] Nov 03 14:13:58.908904 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.908876 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6375e7-7c79699d46-k2kqz"] Nov 03 14:13:58.951890 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:58.951848 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724623c4-14a4-41c5-93b8-9be7c80dcd32" path="/var/lib/kubelet/pods/724623c4-14a4-41c5-93b8-9be7c80dcd32/volumes" Nov 03 14:13:59.655570 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:13:59.655537 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:00.716992 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.716952 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9"] Nov 03 14:14:00.717555 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.717538 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d9a8e841-252f-4396-94cd-32536e18f32c" containerName="revoke-keys" Nov 03 14:14:00.717611 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.717556 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a8e841-252f-4396-94cd-32536e18f32c" containerName="revoke-keys" Nov 03 14:14:00.717611 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.717569 4924 cpu_manager.go:414] "containerMap: removing container" podUID="724623c4-14a4-41c5-93b8-9be7c80dcd32" containerName="maas-api" Nov 03 14:14:00.717611 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.717574 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724623c4-14a4-41c5-93b8-9be7c80dcd32" containerName="maas-api" Nov 03 14:14:00.717727 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.717665 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a8e841-252f-4396-94cd-32536e18f32c" containerName="revoke-keys" Nov 03 14:14:00.717727 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.717675 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="724623c4-14a4-41c5-93b8-9be7c80dcd32" containerName="maas-api" Nov 03 14:14:00.720520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.720498 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.817314 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.817274 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:14:00.817555 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.817342 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:14:00.820805 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.820784 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9"] Nov 03 14:14:00.834380 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.834355 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.834530 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.834408 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.834599 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.834557 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5v6\" (UniqueName: \"kubernetes.io/projected/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-kube-api-access-4s5v6\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.935667 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.935633 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.935836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.935679 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.935836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.935726 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5v6\" (UniqueName: \"kubernetes.io/projected/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-kube-api-access-4s5v6\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.936275 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.936256 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.938105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.938061 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:00.991557 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:00.991480 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5v6\" (UniqueName: \"kubernetes.io/projected/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-kube-api-access-4s5v6\") pod \"maas-api-revoke-sub-fc8590ea-kklz9\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:01.032571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:01.032535 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:01.404243 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:01.404213 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9"] Nov 03 14:14:01.405406 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:14:01.405373 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd867b821_2ea6_4fbf_b7a1_b9e3cf3ebbae.slice/crio-8c74731941674689254acbe778691e20f1b2247e92b59c28ca5e476a9c460007 WatchSource:0}: Error finding container 8c74731941674689254acbe778691e20f1b2247e92b59c28ca5e476a9c460007: Status 404 returned error can't find the container with id 8c74731941674689254acbe778691e20f1b2247e92b59c28ca5e476a9c460007 Nov 03 14:14:01.756899 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:01.756862 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" event={"ID":"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae","Type":"ContainerStarted","Data":"9dc6071890d710aa10deb75c3dcfa75e746c1a1f55c4dacde8642d08b5669423"} Nov 03 14:14:01.757297 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:01.756913 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" event={"ID":"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae","Type":"ContainerStarted","Data":"8c74731941674689254acbe778691e20f1b2247e92b59c28ca5e476a9c460007"} Nov 03 14:14:01.867082 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:01.867014 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" podStartSLOduration=1.8670010449999999 podStartE2EDuration="1.867001045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:01.863653352 +0000 UTC m=+4175.058056946" watchObservedRunningTime="2026-11-03 14:14:01.867001045 +0000 UTC m=+4175.061404639" Nov 03 14:14:02.271714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:02.271674 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:02.790652 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:02.790609 4924 generic.go:356] "Generic (PLEG): container finished" podID="d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae" containerID="9dc6071890d710aa10deb75c3dcfa75e746c1a1f55c4dacde8642d08b5669423" exitCode=0 Nov 03 14:14:02.791209 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:02.790734 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" event={"ID":"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae","Type":"ContainerDied","Data":"9dc6071890d710aa10deb75c3dcfa75e746c1a1f55c4dacde8642d08b5669423"} Nov 03 14:14:02.800779 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:02.800744 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:03.964257 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:03.964225 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:04.094151 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.094047 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-kube-api-access-4s5v6\" (UniqueName: \"kubernetes.io/projected/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-kube-api-access-4s5v6\") pod \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " Nov 03 14:14:04.094151 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.094118 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-openshift-ca\") pod \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " Nov 03 14:14:04.094400 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.094172 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-xks-ca\") pod \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\" (UID: \"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae\") " Nov 03 14:14:04.094788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.094759 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-openshift-ca" pod "d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae" (UID: "d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:14:04.096812 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.096778 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-xks-ca" pod "d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae" (UID: "d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:14:04.096917 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.096868 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-kube-api-access-4s5v6" pod "d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae" (UID: "d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae"). InnerVolumeSpecName "kube-api-access-4s5v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:14:04.195604 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.195564 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5v6\" (UniqueName: \"kubernetes.io/projected/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-kube-api-access-4s5v6\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:04.195604 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.195596 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:04.195604 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.195607 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:04.804325 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.804294 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" Nov 03 14:14:04.804495 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.804293 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9" event={"ID":"d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae","Type":"ContainerDied","Data":"8c74731941674689254acbe778691e20f1b2247e92b59c28ca5e476a9c460007"} Nov 03 14:14:04.804495 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:04.804453 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c74731941674689254acbe778691e20f1b2247e92b59c28ca5e476a9c460007" Nov 03 14:14:05.093270 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:05.093176 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:05.306623 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:05.306583 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:05.440713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:05.440677 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:08.078748 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:08.078666 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:08.219629 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:08.219585 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:10.030299 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.030263 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p"] Nov 03 14:14:10.031059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.031037 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae" containerName="revoke-keys" Nov 03 14:14:10.031131 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.031062 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae" containerName="revoke-keys" Nov 03 14:14:10.031257 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.031246 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae" containerName="revoke-keys" Nov 03 14:14:10.034290 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.034271 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.087494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.087457 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p"] Nov 03 14:14:10.130006 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.129976 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2399b8-serving-cert\"" Nov 03 14:14:10.164890 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.164856 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.165062 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.164988 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb824c1-f17f-42bb-b9e8-d0a783069259-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.165168 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.165144 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlnr\" (UniqueName: \"kubernetes.io/projected/eeb824c1-f17f-42bb-b9e8-d0a783069259-kube-api-access-9xlnr\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.165233 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.165194 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-metrics-certs\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.210571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.210535 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:10.266012 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.265969 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.266235 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.266065 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb824c1-f17f-42bb-b9e8-d0a783069259-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.266235 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.266168 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlnr\" (UniqueName: \"kubernetes.io/projected/eeb824c1-f17f-42bb-b9e8-d0a783069259-kube-api-access-9xlnr\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.266235 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.266205 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-metrics-certs\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.266423 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:14:10.266171 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-serving-cert: secret "maas-api-e2e-inf-a-2399b8-serving-cert" not found Nov 03 14:14:10.267136 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.267108 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb824c1-f17f-42bb-b9e8-d0a783069259-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.267301 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:14:10.267185 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls podName:eeb824c1-f17f-42bb-b9e8-d0a783069259 nodeName:}" failed. No retries permitted until 2026-11-03 14:14:10.7664332 +0000 UTC m=+4183.960836794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls") pod "maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" (UID: "eeb824c1-f17f-42bb-b9e8-d0a783069259") : secret "maas-api-e2e-inf-a-2399b8-serving-cert" not found Nov 03 14:14:10.269896 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.269867 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-metrics-certs\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.321441 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.321357 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlnr\" (UniqueName: \"kubernetes.io/projected/eeb824c1-f17f-42bb-b9e8-d0a783069259-kube-api-access-9xlnr\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.378870 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.378826 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:10.497425 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.497386 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:10.772677 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.772639 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.775259 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.775227 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls\") pod \"maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:10.947979 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:10.947937 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:11.318384 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:11.318353 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p"] Nov 03 14:14:11.319170 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:14:11.319138 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb824c1_f17f_42bb_b9e8_d0a783069259.slice/crio-0830ad76386045abff68ecb161738f5246e552fcca11123ff84177e080e98533 WatchSource:0}: Error finding container 0830ad76386045abff68ecb161738f5246e552fcca11123ff84177e080e98533: Status 404 returned error can't find the container with id 0830ad76386045abff68ecb161738f5246e552fcca11123ff84177e080e98533 Nov 03 14:14:11.855640 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:11.855602 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" event={"ID":"eeb824c1-f17f-42bb-b9e8-d0a783069259","Type":"ContainerStarted","Data":"0830ad76386045abff68ecb161738f5246e552fcca11123ff84177e080e98533"} Nov 03 14:14:12.865597 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:12.865557 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" event={"ID":"eeb824c1-f17f-42bb-b9e8-d0a783069259","Type":"ContainerStarted","Data":"4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232"} Nov 03 14:14:12.866189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:12.865673 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:12.976391 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:12.976329 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" podStartSLOduration=3.059750507 podStartE2EDuration="3.976310078s" totalImagesPullingTime="916.559571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:12.973223709 +0000 UTC m=+4186.167627303" watchObservedRunningTime="2026-11-03 14:14:12.976310078 +0000 UTC m=+4186.170713673" Nov 03 14:14:13.545955 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:13.545918 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:13.662185 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:13.662144 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:15.575145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:15.575110 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:15.981287 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:15.981250 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:16.338501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:16.338416 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:16.586920 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:16.586883 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:16.815092 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:16.808703 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:18.566264 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:14:18.566221 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:14:18.880513 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:18.880445 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:14:20.927497 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:20.927457 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:21.103479 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:21.103448 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:21.416584 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:21.416546 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:21.614755 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:21.614711 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:24.360329 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:24.360300 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:24.547566 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:24.543674 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:26.673921 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:26.673875 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:27.661546 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:27.661513 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:14:27.676229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:27.676202 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:14:31.857318 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:31.857276 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:36.953672 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:36.953630 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:40.177997 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:40.177960 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:40.297083 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:40.297046 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:42.063471 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:42.063441 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:47.273616 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:47.273581 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:48.559820 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:14:48.559777 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:14:51.751433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:51.751397 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:53.561788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:53.561755 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:14:58.713388 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:14:58.713354 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:00.187144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.187056 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78"] Nov 03 14:15:00.209837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.209624 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h"] Nov 03 14:15:00.210138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.209660 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:00.217547 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.217513 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:00.227966 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.227934 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9"] Nov 03 14:15:00.234462 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.234437 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:00.236230 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.236205 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz"] Nov 03 14:15:00.242875 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.242851 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:00.266213 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.266172 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:15:00.274248 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.274215 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz"] Nov 03 14:15:00.280686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.280655 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k"] Nov 03 14:15:00.280830 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.280786 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:00.285690 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.285662 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:00.296528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.296497 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78"] Nov 03 14:15:00.310865 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.310835 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngdx\" (UniqueName: \"kubernetes.io/projected/5d5720f5-f068-4e50-be5a-e4a2f9cae662-kube-api-access-rngdx\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz\" (UID: \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:00.312956 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.312931 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h"] Nov 03 14:15:00.321453 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.321411 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c4285e-17f8-4972-aa11-873c8aff6f8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9\" (UID: \"99c4285e-17f8-4972-aa11-873c8aff6f8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:00.321621 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.321463 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjqk\" (UniqueName: \"kubernetes.io/projected/43f0cc36-c074-42f2-9394-f864905b7187-kube-api-access-qqjqk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78\" (UID: \"43f0cc36-c074-42f2-9394-f864905b7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:00.321621 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.321556 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnt2k\" (UniqueName: \"kubernetes.io/projected/9586e6bb-4bd4-43a6-a283-955053cf2b26-kube-api-access-vnt2k\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h\" (UID: \"9586e6bb-4bd4-43a6-a283-955053cf2b26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:00.321621 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.321607 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f0cc36-c074-42f2-9394-f864905b7187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78\" (UID: \"43f0cc36-c074-42f2-9394-f864905b7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:00.321849 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.321665 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5720f5-f068-4e50-be5a-e4a2f9cae662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz\" (UID: \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:00.321849 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.321736 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9586e6bb-4bd4-43a6-a283-955053cf2b26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h\" (UID: \"9586e6bb-4bd4-43a6-a283-955053cf2b26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:00.321849 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.321824 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npzdn\" (UniqueName: \"kubernetes.io/projected/99c4285e-17f8-4972-aa11-873c8aff6f8f-kube-api-access-npzdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9\" (UID: \"99c4285e-17f8-4972-aa11-873c8aff6f8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:00.370196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.369242 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9"] Nov 03 14:15:00.385008 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.384883 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz"] Nov 03 14:15:00.390110 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.389912 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k"] Nov 03 14:15:00.407550 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.406467 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz"] Nov 03 14:15:00.424000 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.423961 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngdx\" (UniqueName: \"kubernetes.io/projected/5d5720f5-f068-4e50-be5a-e4a2f9cae662-kube-api-access-rngdx\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz\" (UID: \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:00.424323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.424298 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd77h\" (UniqueName: \"kubernetes.io/projected/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-kube-api-access-bd77h\") pod \"maas-api-key-cleanup-29895255-pf94k\" (UID: \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:00.424488 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.424471 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c4285e-17f8-4972-aa11-873c8aff6f8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9\" (UID: \"99c4285e-17f8-4972-aa11-873c8aff6f8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:00.424616 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.424599 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjqk\" (UniqueName: \"kubernetes.io/projected/43f0cc36-c074-42f2-9394-f864905b7187-kube-api-access-qqjqk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78\" (UID: \"43f0cc36-c074-42f2-9394-f864905b7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:00.424785 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.424768 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnt2k\" (UniqueName: \"kubernetes.io/projected/9586e6bb-4bd4-43a6-a283-955053cf2b26-kube-api-access-vnt2k\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h\" (UID: \"9586e6bb-4bd4-43a6-a283-955053cf2b26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:00.424915 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.424900 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f0cc36-c074-42f2-9394-f864905b7187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78\" (UID: \"43f0cc36-c074-42f2-9394-f864905b7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:00.425087 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.425050 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5720f5-f068-4e50-be5a-e4a2f9cae662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz\" (UID: \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:00.427718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.425827 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f0cc36-c074-42f2-9394-f864905b7187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78\" (UID: \"43f0cc36-c074-42f2-9394-f864905b7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:00.427718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.426617 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c4285e-17f8-4972-aa11-873c8aff6f8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9\" (UID: \"99c4285e-17f8-4972-aa11-873c8aff6f8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:00.427718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.427047 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9586e6bb-4bd4-43a6-a283-955053cf2b26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h\" (UID: \"9586e6bb-4bd4-43a6-a283-955053cf2b26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:00.427718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.427654 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5720f5-f068-4e50-be5a-e4a2f9cae662-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz\" (UID: \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:00.429182 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.429157 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895255-pf94k\" (UID: \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:00.429380 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.429361 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npzdn\" (UniqueName: \"kubernetes.io/projected/99c4285e-17f8-4972-aa11-873c8aff6f8f-kube-api-access-npzdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9\" (UID: \"99c4285e-17f8-4972-aa11-873c8aff6f8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:00.429546 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.429527 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f707415-eaee-4191-8deb-2b21e34a4922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz\" (UID: \"9f707415-eaee-4191-8deb-2b21e34a4922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:00.429690 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.429673 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwv2m\" (UniqueName: \"kubernetes.io/projected/9f707415-eaee-4191-8deb-2b21e34a4922-kube-api-access-wwv2m\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz\" (UID: \"9f707415-eaee-4191-8deb-2b21e34a4922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:00.429871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.429843 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9586e6bb-4bd4-43a6-a283-955053cf2b26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h\" (UID: \"9586e6bb-4bd4-43a6-a283-955053cf2b26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:00.478985 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.478882 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngdx\" (UniqueName: \"kubernetes.io/projected/5d5720f5-f068-4e50-be5a-e4a2f9cae662-kube-api-access-rngdx\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz\" (UID: \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:00.483098 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.481421 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjqk\" (UniqueName: \"kubernetes.io/projected/43f0cc36-c074-42f2-9394-f864905b7187-kube-api-access-qqjqk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78\" (UID: \"43f0cc36-c074-42f2-9394-f864905b7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:00.483098 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.481886 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnt2k\" (UniqueName: \"kubernetes.io/projected/9586e6bb-4bd4-43a6-a283-955053cf2b26-kube-api-access-vnt2k\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h\" (UID: \"9586e6bb-4bd4-43a6-a283-955053cf2b26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:00.495064 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.495028 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npzdn\" (UniqueName: \"kubernetes.io/projected/99c4285e-17f8-4972-aa11-873c8aff6f8f-kube-api-access-npzdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9\" (UID: \"99c4285e-17f8-4972-aa11-873c8aff6f8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:00.530861 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.530804 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895255-pf94k\" (UID: \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:00.531187 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.530919 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f707415-eaee-4191-8deb-2b21e34a4922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz\" (UID: \"9f707415-eaee-4191-8deb-2b21e34a4922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:00.531187 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.530967 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwv2m\" (UniqueName: \"kubernetes.io/projected/9f707415-eaee-4191-8deb-2b21e34a4922-kube-api-access-wwv2m\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz\" (UID: \"9f707415-eaee-4191-8deb-2b21e34a4922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:00.531187 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.531023 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd77h\" (UniqueName: \"kubernetes.io/projected/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-kube-api-access-bd77h\") pod \"maas-api-key-cleanup-29895255-pf94k\" (UID: \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:00.536106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.532322 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895255-pf94k\" (UID: \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:00.536106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.532967 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f707415-eaee-4191-8deb-2b21e34a4922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz\" (UID: \"9f707415-eaee-4191-8deb-2b21e34a4922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:00.552662 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.552303 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:00.573738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.563252 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:00.579109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.577610 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:00.618409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.588247 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:00.618409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.592893 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd77h\" (UniqueName: \"kubernetes.io/projected/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-kube-api-access-bd77h\") pod \"maas-api-key-cleanup-29895255-pf94k\" (UID: \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:00.618409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.604480 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwv2m\" (UniqueName: \"kubernetes.io/projected/9f707415-eaee-4191-8deb-2b21e34a4922-kube-api-access-wwv2m\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz\" (UID: \"9f707415-eaee-4191-8deb-2b21e34a4922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:00.642391 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.627235 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:00.915949 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:00.915908 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:01.722096 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:01.708652 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:01.910095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:01.898172 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:02.184804 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:02.184448 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78"] Nov 03 14:15:02.418196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:02.396366 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:15:02.485132 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:02.485029 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" event={"ID":"43f0cc36-c074-42f2-9394-f864905b7187","Type":"ContainerStarted","Data":"69ef1544d51b7ab60dec70785bf924968bb314dba00d09ff78da19ec235620ca"} Nov 03 14:15:02.515965 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:02.515929 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k"] Nov 03 14:15:02.933804 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:02.933768 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz"] Nov 03 14:15:03.026567 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:15:03.026529 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9586e6bb_4bd4_43a6_a283_955053cf2b26.slice/crio-c2c07aaca50262ac785b64e4459cc6da772008968a3e551b56428107f8e662cc WatchSource:0}: Error finding container c2c07aaca50262ac785b64e4459cc6da772008968a3e551b56428107f8e662cc: Status 404 returned error can't find the container with id c2c07aaca50262ac785b64e4459cc6da772008968a3e551b56428107f8e662cc Nov 03 14:15:03.026737 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.026664 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h"] Nov 03 14:15:03.035002 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.034978 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9"] Nov 03 14:15:03.041547 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:15:03.041518 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c4285e_17f8_4972_aa11_873c8aff6f8f.slice/crio-29443d8919f35328115f966e70ec2feaf30880e5f5829d23649f6e6a69c5f628 WatchSource:0}: Error finding container 29443d8919f35328115f966e70ec2feaf30880e5f5829d23649f6e6a69c5f628: Status 404 returned error can't find the container with id 29443d8919f35328115f966e70ec2feaf30880e5f5829d23649f6e6a69c5f628 Nov 03 14:15:03.199816 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.199776 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz"] Nov 03 14:15:03.200719 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:15:03.200693 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f707415_eaee_4191_8deb_2b21e34a4922.slice/crio-eba6dd7138580410b417913abeb58878e688f4ac8f8ce9c67dd7af02a5c4c166 WatchSource:0}: Error finding container eba6dd7138580410b417913abeb58878e688f4ac8f8ce9c67dd7af02a5c4c166: Status 404 returned error can't find the container with id eba6dd7138580410b417913abeb58878e688f4ac8f8ce9c67dd7af02a5c4c166 Nov 03 14:15:03.398163 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.398130 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:03.493596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.493556 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" event={"ID":"9f707415-eaee-4191-8deb-2b21e34a4922","Type":"ContainerStarted","Data":"eba6dd7138580410b417913abeb58878e688f4ac8f8ce9c67dd7af02a5c4c166"} Nov 03 14:15:03.495386 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.495346 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" event={"ID":"99c4285e-17f8-4972-aa11-873c8aff6f8f","Type":"ContainerStarted","Data":"6d7707b0c65c1e8eef987a7ffa0579ad0e97b50251fd9da4e7d3ba3aa51df023"} Nov 03 14:15:03.495531 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.495402 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" event={"ID":"99c4285e-17f8-4972-aa11-873c8aff6f8f","Type":"ContainerStarted","Data":"29443d8919f35328115f966e70ec2feaf30880e5f5829d23649f6e6a69c5f628"} Nov 03 14:15:03.497139 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.497107 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" event={"ID":"9586e6bb-4bd4-43a6-a283-955053cf2b26","Type":"ContainerStarted","Data":"13389cac8b07332776bfb200dfeb352fb2a22d558d1c612cf66071ba6ce180a6"} Nov 03 14:15:03.497257 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.497157 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" event={"ID":"9586e6bb-4bd4-43a6-a283-955053cf2b26","Type":"ContainerStarted","Data":"c2c07aaca50262ac785b64e4459cc6da772008968a3e551b56428107f8e662cc"} Nov 03 14:15:03.498735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.498704 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" event={"ID":"95687e27-c3ff-4a61-8a3c-1eaeceac80e4","Type":"ContainerStarted","Data":"ac2ddccccc551e1a731d81c926c3ac9e42ed21eeeee96d51bd56cb72a429bf87"} Nov 03 14:15:03.498838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.498747 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" event={"ID":"95687e27-c3ff-4a61-8a3c-1eaeceac80e4","Type":"ContainerStarted","Data":"b37fec4edc383393fa7021e8fbf0c225f39890e85cbfb5e7502a0fa1606cbe78"} Nov 03 14:15:03.500549 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.500519 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" event={"ID":"5d5720f5-f068-4e50-be5a-e4a2f9cae662","Type":"ContainerStarted","Data":"9c536357ed10abf7d228031522677aab955e54486195a58b17a9142c293b4e80"} Nov 03 14:15:03.500642 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.500558 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" event={"ID":"5d5720f5-f068-4e50-be5a-e4a2f9cae662","Type":"ContainerStarted","Data":"14302dc9ce388a2ed428a6f4b288c11f29f95dbb9394420311fe1b9ff3624e53"} Nov 03 14:15:03.502111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.502088 4924 generic.go:356] "Generic (PLEG): container finished" podID="43f0cc36-c074-42f2-9394-f864905b7187" containerID="7bc8b751ab9bd16b727d6ae2ee854cc02b153a16c8c97aa7ce53d79aca0b719f" exitCode=60 Nov 03 14:15:03.502200 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.502120 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" event={"ID":"43f0cc36-c074-42f2-9394-f864905b7187","Type":"ContainerDied","Data":"7bc8b751ab9bd16b727d6ae2ee854cc02b153a16c8c97aa7ce53d79aca0b719f"} Nov 03 14:15:03.502327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.502314 4924 scope.go:122] "RemoveContainer" containerID="7bc8b751ab9bd16b727d6ae2ee854cc02b153a16c8c97aa7ce53d79aca0b719f" Nov 03 14:15:03.631899 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.631826 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" podStartSLOduration=3.631805291 podStartE2EDuration="3.631805291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:03.6306369 +0000 UTC m=+4236.825040497" watchObservedRunningTime="2026-11-03 14:15:03.631805291 +0000 UTC m=+4236.826208886" Nov 03 14:15:03.747729 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.747655 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" podStartSLOduration=3.747636836 podStartE2EDuration="3.747636836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:03.746249073 +0000 UTC m=+4236.940652666" watchObservedRunningTime="2026-11-03 14:15:03.747636836 +0000 UTC m=+4236.942040431" Nov 03 14:15:03.924776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.924741 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j"] Nov 03 14:15:03.924990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.924953 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" containerID="cri-o://0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761" gracePeriod=30 Nov 03 14:15:03.946735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.946284 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:03.968693 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:03.968642 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" podStartSLOduration=3.968629563 podStartE2EDuration="3.968629563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:03.965129646 +0000 UTC m=+4237.159533234" watchObservedRunningTime="2026-11-03 14:15:03.968629563 +0000 UTC m=+4237.163033150" Nov 03 14:15:04.108092 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.095587 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" podStartSLOduration=4.09556501 podStartE2EDuration="4.09556501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:04.083354427 +0000 UTC m=+4237.277758022" watchObservedRunningTime="2026-11-03 14:15:04.09556501 +0000 UTC m=+4237.289968605" Nov 03 14:15:04.436347 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.436308 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:04.513059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.512966 4924 generic.go:356] "Generic (PLEG): container finished" podID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerID="13389cac8b07332776bfb200dfeb352fb2a22d558d1c612cf66071ba6ce180a6" exitCode=60 Nov 03 14:15:04.513276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.513045 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" event={"ID":"9586e6bb-4bd4-43a6-a283-955053cf2b26","Type":"ContainerDied","Data":"13389cac8b07332776bfb200dfeb352fb2a22d558d1c612cf66071ba6ce180a6"} Nov 03 14:15:04.513388 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.513374 4924 scope.go:122] "RemoveContainer" containerID="13389cac8b07332776bfb200dfeb352fb2a22d558d1c612cf66071ba6ce180a6" Nov 03 14:15:04.514904 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.514882 4924 generic.go:356] "Generic (PLEG): container finished" podID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerID="ac2ddccccc551e1a731d81c926c3ac9e42ed21eeeee96d51bd56cb72a429bf87" exitCode=60 Nov 03 14:15:04.515025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.514968 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" event={"ID":"95687e27-c3ff-4a61-8a3c-1eaeceac80e4","Type":"ContainerDied","Data":"ac2ddccccc551e1a731d81c926c3ac9e42ed21eeeee96d51bd56cb72a429bf87"} Nov 03 14:15:04.515326 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.515252 4924 scope.go:122] "RemoveContainer" containerID="ac2ddccccc551e1a731d81c926c3ac9e42ed21eeeee96d51bd56cb72a429bf87" Nov 03 14:15:04.516721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.516703 4924 generic.go:356] "Generic (PLEG): container finished" podID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerID="9c536357ed10abf7d228031522677aab955e54486195a58b17a9142c293b4e80" exitCode=60 Nov 03 14:15:04.516887 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.516788 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" event={"ID":"5d5720f5-f068-4e50-be5a-e4a2f9cae662","Type":"ContainerDied","Data":"9c536357ed10abf7d228031522677aab955e54486195a58b17a9142c293b4e80"} Nov 03 14:15:04.516973 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.516958 4924 scope.go:122] "RemoveContainer" containerID="9c536357ed10abf7d228031522677aab955e54486195a58b17a9142c293b4e80" Nov 03 14:15:04.519474 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.519454 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" event={"ID":"43f0cc36-c074-42f2-9394-f864905b7187","Type":"ContainerStarted","Data":"c488b447af50756e8db8bc5930133eadc88c717919c8e9da59df65d27bbb051a"} Nov 03 14:15:04.520570 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.520548 4924 generic.go:356] "Generic (PLEG): container finished" podID="9f707415-eaee-4191-8deb-2b21e34a4922" containerID="ce28864622a229bbdc14ee06f94b5c75f2c58701db813fb6189bbbca0f71cdd5" exitCode=60 Nov 03 14:15:04.520665 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.520630 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" event={"ID":"9f707415-eaee-4191-8deb-2b21e34a4922","Type":"ContainerDied","Data":"ce28864622a229bbdc14ee06f94b5c75f2c58701db813fb6189bbbca0f71cdd5"} Nov 03 14:15:04.520796 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.520783 4924 scope.go:122] "RemoveContainer" containerID="ce28864622a229bbdc14ee06f94b5c75f2c58701db813fb6189bbbca0f71cdd5" Nov 03 14:15:04.522241 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.522220 4924 generic.go:356] "Generic (PLEG): container finished" podID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerID="6d7707b0c65c1e8eef987a7ffa0579ad0e97b50251fd9da4e7d3ba3aa51df023" exitCode=60 Nov 03 14:15:04.522328 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.522284 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" event={"ID":"99c4285e-17f8-4972-aa11-873c8aff6f8f","Type":"ContainerDied","Data":"6d7707b0c65c1e8eef987a7ffa0579ad0e97b50251fd9da4e7d3ba3aa51df023"} Nov 03 14:15:04.522440 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.522429 4924 scope.go:122] "RemoveContainer" containerID="6d7707b0c65c1e8eef987a7ffa0579ad0e97b50251fd9da4e7d3ba3aa51df023" Nov 03 14:15:04.836678 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:04.836628 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" podStartSLOduration=4.8366129 podStartE2EDuration="4.8366129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:04.835616028 +0000 UTC m=+4238.030019622" watchObservedRunningTime="2026-11-03 14:15:04.8366129 +0000 UTC m=+4238.031016514" Nov 03 14:15:05.535509 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.535470 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" event={"ID":"9586e6bb-4bd4-43a6-a283-955053cf2b26","Type":"ContainerStarted","Data":"21f350d2225db26a9f630edbbf8e05c59228a82b2b591444042e12702459b2b0"} Nov 03 14:15:05.538708 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.538671 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" event={"ID":"95687e27-c3ff-4a61-8a3c-1eaeceac80e4","Type":"ContainerStarted","Data":"387ea723a65adbf848916793bb029806ebd67f9a0270c0e35b171bc9de997809"} Nov 03 14:15:05.540944 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.540915 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" event={"ID":"5d5720f5-f068-4e50-be5a-e4a2f9cae662","Type":"ContainerStarted","Data":"85e584911204439d37d78ede5116d5451d5f6f162dc9d3d935e39856838da11b"} Nov 03 14:15:05.543033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.543007 4924 generic.go:356] "Generic (PLEG): container finished" podID="43f0cc36-c074-42f2-9394-f864905b7187" containerID="c488b447af50756e8db8bc5930133eadc88c717919c8e9da59df65d27bbb051a" exitCode=60 Nov 03 14:15:05.543130 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.543098 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" event={"ID":"43f0cc36-c074-42f2-9394-f864905b7187","Type":"ContainerDied","Data":"c488b447af50756e8db8bc5930133eadc88c717919c8e9da59df65d27bbb051a"} Nov 03 14:15:05.543200 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.543155 4924 scope.go:122] "RemoveContainer" containerID="7bc8b751ab9bd16b727d6ae2ee854cc02b153a16c8c97aa7ce53d79aca0b719f" Nov 03 14:15:05.543331 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.543312 4924 scope.go:122] "RemoveContainer" containerID="c488b447af50756e8db8bc5930133eadc88c717919c8e9da59df65d27bbb051a" Nov 03 14:15:05.543584 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:05.543552 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78_odh-ai-gateway-infra(43f0cc36-c074-42f2-9394-f864905b7187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" podUID="43f0cc36-c074-42f2-9394-f864905b7187" Nov 03 14:15:05.545120 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.545097 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" event={"ID":"9f707415-eaee-4191-8deb-2b21e34a4922","Type":"ContainerStarted","Data":"65b7bc0c33f280f49035dc5ac11ab81d19f970035b34a73111ba4ff4a28a9690"} Nov 03 14:15:05.546996 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.546971 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" event={"ID":"99c4285e-17f8-4972-aa11-873c8aff6f8f","Type":"ContainerStarted","Data":"cb130c95a05d11df28120317f06e636c445d98f2c5649448f45d915660e7daec"} Nov 03 14:15:05.547179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.547163 4924 scope.go:122] "RemoveContainer" containerID="cb130c95a05d11df28120317f06e636c445d98f2c5649448f45d915660e7daec" Nov 03 14:15:05.547384 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:05.547367 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9_odh-ai-gateway-infra(99c4285e-17f8-4972-aa11-873c8aff6f8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" Nov 03 14:15:05.783143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:05.782935 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" podStartSLOduration=5.782920381 podStartE2EDuration="5.782920381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:05.782066174 +0000 UTC m=+4238.976469767" watchObservedRunningTime="2026-11-03 14:15:05.782920381 +0000 UTC m=+4238.977323975" Nov 03 14:15:06.344760 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.344723 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:06.498540 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.498507 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:06.559610 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.559577 4924 generic.go:356] "Generic (PLEG): container finished" podID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerID="21f350d2225db26a9f630edbbf8e05c59228a82b2b591444042e12702459b2b0" exitCode=60 Nov 03 14:15:06.560110 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.559670 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" event={"ID":"9586e6bb-4bd4-43a6-a283-955053cf2b26","Type":"ContainerDied","Data":"21f350d2225db26a9f630edbbf8e05c59228a82b2b591444042e12702459b2b0"} Nov 03 14:15:06.560110 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.559740 4924 scope.go:122] "RemoveContainer" containerID="13389cac8b07332776bfb200dfeb352fb2a22d558d1c612cf66071ba6ce180a6" Nov 03 14:15:06.560110 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.559957 4924 scope.go:122] "RemoveContainer" containerID="21f350d2225db26a9f630edbbf8e05c59228a82b2b591444042e12702459b2b0" Nov 03 14:15:06.560394 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:06.560368 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h_odh-ai-gateway-infra(9586e6bb-4bd4-43a6-a283-955053cf2b26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" Nov 03 14:15:06.561831 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.561801 4924 generic.go:356] "Generic (PLEG): container finished" podID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerID="387ea723a65adbf848916793bb029806ebd67f9a0270c0e35b171bc9de997809" exitCode=60 Nov 03 14:15:06.561913 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.561883 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" event={"ID":"95687e27-c3ff-4a61-8a3c-1eaeceac80e4","Type":"ContainerDied","Data":"387ea723a65adbf848916793bb029806ebd67f9a0270c0e35b171bc9de997809"} Nov 03 14:15:06.562163 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.562145 4924 scope.go:122] "RemoveContainer" containerID="387ea723a65adbf848916793bb029806ebd67f9a0270c0e35b171bc9de997809" Nov 03 14:15:06.562457 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:06.562437 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895255-pf94k_odh-ai-gateway-infra(95687e27-c3ff-4a61-8a3c-1eaeceac80e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" Nov 03 14:15:06.563624 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.563605 4924 generic.go:356] "Generic (PLEG): container finished" podID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerID="85e584911204439d37d78ede5116d5451d5f6f162dc9d3d935e39856838da11b" exitCode=60 Nov 03 14:15:06.563707 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.563681 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" event={"ID":"5d5720f5-f068-4e50-be5a-e4a2f9cae662","Type":"ContainerDied","Data":"85e584911204439d37d78ede5116d5451d5f6f162dc9d3d935e39856838da11b"} Nov 03 14:15:06.563908 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.563892 4924 scope.go:122] "RemoveContainer" containerID="85e584911204439d37d78ede5116d5451d5f6f162dc9d3d935e39856838da11b" Nov 03 14:15:06.564175 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:06.564151 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz_odh-ai-gateway-infra(5d5720f5-f068-4e50-be5a-e4a2f9cae662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" Nov 03 14:15:06.565438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.565415 4924 scope.go:122] "RemoveContainer" containerID="c488b447af50756e8db8bc5930133eadc88c717919c8e9da59df65d27bbb051a" Nov 03 14:15:06.565670 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:06.565650 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78_odh-ai-gateway-infra(43f0cc36-c074-42f2-9394-f864905b7187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" podUID="43f0cc36-c074-42f2-9394-f864905b7187" Nov 03 14:15:06.567050 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.567029 4924 generic.go:356] "Generic (PLEG): container finished" podID="9f707415-eaee-4191-8deb-2b21e34a4922" containerID="65b7bc0c33f280f49035dc5ac11ab81d19f970035b34a73111ba4ff4a28a9690" exitCode=60 Nov 03 14:15:06.567162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.567097 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" event={"ID":"9f707415-eaee-4191-8deb-2b21e34a4922","Type":"ContainerDied","Data":"65b7bc0c33f280f49035dc5ac11ab81d19f970035b34a73111ba4ff4a28a9690"} Nov 03 14:15:06.567301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.567286 4924 scope.go:122] "RemoveContainer" containerID="65b7bc0c33f280f49035dc5ac11ab81d19f970035b34a73111ba4ff4a28a9690" Nov 03 14:15:06.567527 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:06.567504 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz_odh-ai-gateway-infra(9f707415-eaee-4191-8deb-2b21e34a4922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" Nov 03 14:15:06.569015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.568993 4924 generic.go:356] "Generic (PLEG): container finished" podID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerID="cb130c95a05d11df28120317f06e636c445d98f2c5649448f45d915660e7daec" exitCode=60 Nov 03 14:15:06.569147 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.569020 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" event={"ID":"99c4285e-17f8-4972-aa11-873c8aff6f8f","Type":"ContainerDied","Data":"cb130c95a05d11df28120317f06e636c445d98f2c5649448f45d915660e7daec"} Nov 03 14:15:06.569236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.569221 4924 scope.go:122] "RemoveContainer" containerID="cb130c95a05d11df28120317f06e636c445d98f2c5649448f45d915660e7daec" Nov 03 14:15:06.569423 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:06.569404 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9_odh-ai-gateway-infra(99c4285e-17f8-4972-aa11-873c8aff6f8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" Nov 03 14:15:06.574335 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.574313 4924 scope.go:122] "RemoveContainer" containerID="ac2ddccccc551e1a731d81c926c3ac9e42ed21eeeee96d51bd56cb72a429bf87" Nov 03 14:15:06.587306 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.587286 4924 scope.go:122] "RemoveContainer" containerID="9c536357ed10abf7d228031522677aab955e54486195a58b17a9142c293b4e80" Nov 03 14:15:06.600643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.600623 4924 scope.go:122] "RemoveContainer" containerID="ce28864622a229bbdc14ee06f94b5c75f2c58701db813fb6189bbbca0f71cdd5" Nov 03 14:15:06.613144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.613126 4924 scope.go:122] "RemoveContainer" containerID="6d7707b0c65c1e8eef987a7ffa0579ad0e97b50251fd9da4e7d3ba3aa51df023" Nov 03 14:15:06.808907 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.808845 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:06.940543 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:06.940493 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:07.099858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:07.099808 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:07.578632 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:07.578601 4924 scope.go:122] "RemoveContainer" containerID="387ea723a65adbf848916793bb029806ebd67f9a0270c0e35b171bc9de997809" Nov 03 14:15:07.579116 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:07.578954 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895255-pf94k_odh-ai-gateway-infra(95687e27-c3ff-4a61-8a3c-1eaeceac80e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" Nov 03 14:15:07.580307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:07.580282 4924 scope.go:122] "RemoveContainer" containerID="85e584911204439d37d78ede5116d5451d5f6f162dc9d3d935e39856838da11b" Nov 03 14:15:07.580567 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:07.580546 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz_odh-ai-gateway-infra(5d5720f5-f068-4e50-be5a-e4a2f9cae662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" Nov 03 14:15:07.581960 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:07.581935 4924 scope.go:122] "RemoveContainer" containerID="65b7bc0c33f280f49035dc5ac11ab81d19f970035b34a73111ba4ff4a28a9690" Nov 03 14:15:07.582210 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:07.582191 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz_odh-ai-gateway-infra(9f707415-eaee-4191-8deb-2b21e34a4922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" Nov 03 14:15:07.583624 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:07.583605 4924 scope.go:122] "RemoveContainer" containerID="cb130c95a05d11df28120317f06e636c445d98f2c5649448f45d915660e7daec" Nov 03 14:15:07.583845 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:07.583822 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9_odh-ai-gateway-infra(99c4285e-17f8-4972-aa11-873c8aff6f8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" Nov 03 14:15:07.585320 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:07.585304 4924 scope.go:122] "RemoveContainer" containerID="21f350d2225db26a9f630edbbf8e05c59228a82b2b591444042e12702459b2b0" Nov 03 14:15:07.585558 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:07.585537 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h_odh-ai-gateway-infra(9586e6bb-4bd4-43a6-a283-955053cf2b26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" Nov 03 14:15:07.804521 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:07.804486 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:09.203959 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.203898 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:15:09.313625 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313587 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-socket\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.313811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313635 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-token\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-token\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.313811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313667 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-envoy\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.313811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313693 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-kube-api-access-z7qf9\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-kube-api-access-z7qf9\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.313811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313749 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-data\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.313811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313779 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-credential-socket\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.313811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313796 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-certs\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.314177 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313901 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istiod-ca-cert\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.314177 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313901 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-socket" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:09.314177 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.313949 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-podinfo\") pod \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\" (UID: \"081b76f0-1d17-41ed-b0ba-9256d5bd919c\") " Nov 03 14:15:09.314177 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.314166 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-credential-socket" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:09.314387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.314182 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-data" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:09.314515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.314488 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-certs" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:09.314598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.314516 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istiod-ca-cert" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:09.314653 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.314632 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.314710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.314656 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.314710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.314672 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.314710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.314686 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.316154 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.316123 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-kube-api-access-z7qf9" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "kube-api-access-z7qf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:09.316264 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.316156 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-envoy" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:15:09.316264 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.316193 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-token" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:09.316392 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.316310 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-podinfo" pod "081b76f0-1d17-41ed-b0ba-9256d5bd919c" (UID: "081b76f0-1d17-41ed-b0ba-9256d5bd919c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:15:09.416251 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.416211 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.416437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.416270 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.416437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.416286 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.416437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.416299 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/081b76f0-1d17-41ed-b0ba-9256d5bd919c-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.416437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.416314 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qf9\" (UniqueName: \"kubernetes.io/projected/081b76f0-1d17-41ed-b0ba-9256d5bd919c-kube-api-access-z7qf9\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:09.603343 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.603253 4924 generic.go:356] "Generic (PLEG): container finished" podID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerID="0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761" exitCode=0 Nov 03 14:15:09.603343 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.603314 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" Nov 03 14:15:09.603516 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.603333 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" event={"ID":"081b76f0-1d17-41ed-b0ba-9256d5bd919c","Type":"ContainerDied","Data":"0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761"} Nov 03 14:15:09.603516 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.603390 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j" event={"ID":"081b76f0-1d17-41ed-b0ba-9256d5bd919c","Type":"ContainerDied","Data":"4e8bccb76d3bb94397a9cf2e46d6895a1968209a10f398bcece95cd5b7eb202f"} Nov 03 14:15:09.603516 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.603414 4924 scope.go:122] "RemoveContainer" containerID="0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761" Nov 03 14:15:09.614593 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.614575 4924 scope.go:122] "RemoveContainer" containerID="6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139" Nov 03 14:15:09.624137 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.624116 4924 scope.go:122] "RemoveContainer" containerID="0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761" Nov 03 14:15:09.624417 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:09.624388 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761\": container with ID starting with 0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761 not found: ID does not exist" containerID="0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761" Nov 03 14:15:09.624468 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.624426 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761"} err="failed to get container status \"0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761\": rpc error: code = NotFound desc = could not find container \"0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761\": container with ID starting with 0834c70423cf6ab9c9d1e287e69cf56debe7f37e6468f1788ce2114f74cb5761 not found: ID does not exist" Nov 03 14:15:09.624468 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.624446 4924 scope.go:122] "RemoveContainer" containerID="6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139" Nov 03 14:15:09.624698 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:09.624684 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139\": container with ID starting with 6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139 not found: ID does not exist" containerID="6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139" Nov 03 14:15:09.624750 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.624702 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139"} err="failed to get container status \"6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139\": rpc error: code = NotFound desc = could not find container \"6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139\": container with ID starting with 6be64001732c519b47e5ac84e09e7d74edd7efe1d8909fd127b813c224a25139 not found: ID does not exist" Nov 03 14:15:09.759527 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.759492 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j"] Nov 03 14:15:09.762788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.762760 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b6c8c7-openshift-default-5cc7dc8b7c-m8s8j"] Nov 03 14:15:09.919410 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:09.919380 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:10.954326 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:10.954285 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" path="/var/lib/kubelet/pods/081b76f0-1d17-41ed-b0ba-9256d5bd919c/volumes" Nov 03 14:15:12.807302 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:12.807264 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:12.942929 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:12.942887 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:15.218262 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:15.218227 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:16.948656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:16.948627 4924 scope.go:122] "RemoveContainer" containerID="65b7bc0c33f280f49035dc5ac11ab81d19f970035b34a73111ba4ff4a28a9690" Nov 03 14:15:16.949055 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:16.948726 4924 scope.go:122] "RemoveContainer" containerID="c488b447af50756e8db8bc5930133eadc88c717919c8e9da59df65d27bbb051a" Nov 03 14:15:17.663846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:17.663813 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" event={"ID":"9f707415-eaee-4191-8deb-2b21e34a4922","Type":"ContainerStarted","Data":"66068cc48b8f8ca4e267a4ea2aaafbc341063485b540946943cdf74e12f5085e"} Nov 03 14:15:17.665803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:17.665779 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" event={"ID":"43f0cc36-c074-42f2-9394-f864905b7187","Type":"ContainerStarted","Data":"4e33d21b89d3c9c2dc8d49594e253ad5928a9a41be3bfccf520725179f5548c9"} Nov 03 14:15:17.919592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:17.919487 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:17.944025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:17.943991 4924 scope.go:122] "RemoveContainer" containerID="21f350d2225db26a9f630edbbf8e05c59228a82b2b591444042e12702459b2b0" Nov 03 14:15:17.944236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:17.944095 4924 scope.go:122] "RemoveContainer" containerID="cb130c95a05d11df28120317f06e636c445d98f2c5649448f45d915660e7daec" Nov 03 14:15:18.558764 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:18.558681 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:15:18.674419 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.674381 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" event={"ID":"9586e6bb-4bd4-43a6-a283-955053cf2b26","Type":"ContainerStarted","Data":"9f86fbb790bc0c14f5654b8769df34a0142fed6e38a71b3bb0fc25bbdd62fc17"} Nov 03 14:15:18.676485 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.676452 4924 generic.go:356] "Generic (PLEG): container finished" podID="43f0cc36-c074-42f2-9394-f864905b7187" containerID="4e33d21b89d3c9c2dc8d49594e253ad5928a9a41be3bfccf520725179f5548c9" exitCode=60 Nov 03 14:15:18.676600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.676530 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" event={"ID":"43f0cc36-c074-42f2-9394-f864905b7187","Type":"ContainerDied","Data":"4e33d21b89d3c9c2dc8d49594e253ad5928a9a41be3bfccf520725179f5548c9"} Nov 03 14:15:18.676600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.676589 4924 scope.go:122] "RemoveContainer" containerID="c488b447af50756e8db8bc5930133eadc88c717919c8e9da59df65d27bbb051a" Nov 03 14:15:18.676922 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.676905 4924 scope.go:122] "RemoveContainer" containerID="4e33d21b89d3c9c2dc8d49594e253ad5928a9a41be3bfccf520725179f5548c9" Nov 03 14:15:18.677201 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:18.677178 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78_odh-ai-gateway-infra(43f0cc36-c074-42f2-9394-f864905b7187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" podUID="43f0cc36-c074-42f2-9394-f864905b7187" Nov 03 14:15:18.678731 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.678710 4924 generic.go:356] "Generic (PLEG): container finished" podID="9f707415-eaee-4191-8deb-2b21e34a4922" containerID="66068cc48b8f8ca4e267a4ea2aaafbc341063485b540946943cdf74e12f5085e" exitCode=60 Nov 03 14:15:18.678820 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.678759 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" event={"ID":"9f707415-eaee-4191-8deb-2b21e34a4922","Type":"ContainerDied","Data":"66068cc48b8f8ca4e267a4ea2aaafbc341063485b540946943cdf74e12f5085e"} Nov 03 14:15:18.679090 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.679047 4924 scope.go:122] "RemoveContainer" containerID="66068cc48b8f8ca4e267a4ea2aaafbc341063485b540946943cdf74e12f5085e" Nov 03 14:15:18.679304 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:18.679279 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz_odh-ai-gateway-infra(9f707415-eaee-4191-8deb-2b21e34a4922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" Nov 03 14:15:18.680629 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.680603 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" event={"ID":"99c4285e-17f8-4972-aa11-873c8aff6f8f","Type":"ContainerStarted","Data":"68ddaeb861001bc6faec0cdf5ef34bf21482695fe291de1af20e89d9a1d0147f"} Nov 03 14:15:18.690252 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.690230 4924 scope.go:122] "RemoveContainer" containerID="65b7bc0c33f280f49035dc5ac11ab81d19f970035b34a73111ba4ff4a28a9690" Nov 03 14:15:18.791658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.791624 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz"] Nov 03 14:15:18.891301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.891266 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78"] Nov 03 14:15:18.943274 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:18.943239 4924 scope.go:122] "RemoveContainer" containerID="85e584911204439d37d78ede5116d5451d5f6f162dc9d3d935e39856838da11b" Nov 03 14:15:19.691133 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.691064 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" event={"ID":"5d5720f5-f068-4e50-be5a-e4a2f9cae662","Type":"ContainerStarted","Data":"ae87c70289c802642e10b107a03732e52bc037855f23c90e89c7d39af0b334d7"} Nov 03 14:15:19.697202 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.697172 4924 generic.go:356] "Generic (PLEG): container finished" podID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerID="68ddaeb861001bc6faec0cdf5ef34bf21482695fe291de1af20e89d9a1d0147f" exitCode=60 Nov 03 14:15:19.697312 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.697244 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" event={"ID":"99c4285e-17f8-4972-aa11-873c8aff6f8f","Type":"ContainerDied","Data":"68ddaeb861001bc6faec0cdf5ef34bf21482695fe291de1af20e89d9a1d0147f"} Nov 03 14:15:19.697376 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.697314 4924 scope.go:122] "RemoveContainer" containerID="cb130c95a05d11df28120317f06e636c445d98f2c5649448f45d915660e7daec" Nov 03 14:15:19.697694 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.697669 4924 scope.go:122] "RemoveContainer" containerID="68ddaeb861001bc6faec0cdf5ef34bf21482695fe291de1af20e89d9a1d0147f" Nov 03 14:15:19.698036 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:19.698015 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9_odh-ai-gateway-infra(99c4285e-17f8-4972-aa11-873c8aff6f8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" Nov 03 14:15:19.700208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.700182 4924 generic.go:356] "Generic (PLEG): container finished" podID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerID="9f86fbb790bc0c14f5654b8769df34a0142fed6e38a71b3bb0fc25bbdd62fc17" exitCode=60 Nov 03 14:15:19.700323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.700269 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" event={"ID":"9586e6bb-4bd4-43a6-a283-955053cf2b26","Type":"ContainerDied","Data":"9f86fbb790bc0c14f5654b8769df34a0142fed6e38a71b3bb0fc25bbdd62fc17"} Nov 03 14:15:19.700611 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.700597 4924 scope.go:122] "RemoveContainer" containerID="9f86fbb790bc0c14f5654b8769df34a0142fed6e38a71b3bb0fc25bbdd62fc17" Nov 03 14:15:19.700857 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:19.700832 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h_odh-ai-gateway-infra(9586e6bb-4bd4-43a6-a283-955053cf2b26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" Nov 03 14:15:19.714334 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.713205 4924 scope.go:122] "RemoveContainer" containerID="21f350d2225db26a9f630edbbf8e05c59228a82b2b591444042e12702459b2b0" Nov 03 14:15:19.810334 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.810302 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h"] Nov 03 14:15:19.860995 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.860966 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:19.900721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.900694 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:19.944200 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.944143 4924 scope.go:122] "RemoveContainer" containerID="387ea723a65adbf848916793bb029806ebd67f9a0270c0e35b171bc9de997809" Nov 03 14:15:19.954737 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.954716 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f707415-eaee-4191-8deb-2b21e34a4922-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f707415-eaee-4191-8deb-2b21e34a4922-maas-api-service-ca\") pod \"9f707415-eaee-4191-8deb-2b21e34a4922\" (UID: \"9f707415-eaee-4191-8deb-2b21e34a4922\") " Nov 03 14:15:19.954846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.954833 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f707415-eaee-4191-8deb-2b21e34a4922-kube-api-access-wwv2m\" (UniqueName: \"kubernetes.io/projected/9f707415-eaee-4191-8deb-2b21e34a4922-kube-api-access-wwv2m\") pod \"9f707415-eaee-4191-8deb-2b21e34a4922\" (UID: \"9f707415-eaee-4191-8deb-2b21e34a4922\") " Nov 03 14:15:19.955111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.955062 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f707415-eaee-4191-8deb-2b21e34a4922-maas-api-service-ca" pod "9f707415-eaee-4191-8deb-2b21e34a4922" (UID: "9f707415-eaee-4191-8deb-2b21e34a4922"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:19.955306 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.955269 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f707415-eaee-4191-8deb-2b21e34a4922-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.957018 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:19.956994 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f707415-eaee-4191-8deb-2b21e34a4922-kube-api-access-wwv2m" pod "9f707415-eaee-4191-8deb-2b21e34a4922" (UID: "9f707415-eaee-4191-8deb-2b21e34a4922"). InnerVolumeSpecName "kube-api-access-wwv2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:20.056293 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.056262 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f0cc36-c074-42f2-9394-f864905b7187-kube-api-access-qqjqk\" (UniqueName: \"kubernetes.io/projected/43f0cc36-c074-42f2-9394-f864905b7187-kube-api-access-qqjqk\") pod \"43f0cc36-c074-42f2-9394-f864905b7187\" (UID: \"43f0cc36-c074-42f2-9394-f864905b7187\") " Nov 03 14:15:20.056466 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.056298 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f0cc36-c074-42f2-9394-f864905b7187-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f0cc36-c074-42f2-9394-f864905b7187-maas-api-service-ca\") pod \"43f0cc36-c074-42f2-9394-f864905b7187\" (UID: \"43f0cc36-c074-42f2-9394-f864905b7187\") " Nov 03 14:15:20.056697 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.056678 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwv2m\" (UniqueName: \"kubernetes.io/projected/9f707415-eaee-4191-8deb-2b21e34a4922-kube-api-access-wwv2m\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:20.056838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.056700 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f0cc36-c074-42f2-9394-f864905b7187-maas-api-service-ca" pod "43f0cc36-c074-42f2-9394-f864905b7187" (UID: "43f0cc36-c074-42f2-9394-f864905b7187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:20.058526 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.058503 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f0cc36-c074-42f2-9394-f864905b7187-kube-api-access-qqjqk" pod "43f0cc36-c074-42f2-9394-f864905b7187" (UID: "43f0cc36-c074-42f2-9394-f864905b7187"). InnerVolumeSpecName "kube-api-access-qqjqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:20.132020 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.129387 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9"] Nov 03 14:15:20.158135 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.158103 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjqk\" (UniqueName: \"kubernetes.io/projected/43f0cc36-c074-42f2-9394-f864905b7187-kube-api-access-qqjqk\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:20.158135 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.158135 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f0cc36-c074-42f2-9394-f864905b7187-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:20.273441 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.273405 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:20.708608 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.708574 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" Nov 03 14:15:20.708608 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.708587 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78" event={"ID":"43f0cc36-c074-42f2-9394-f864905b7187","Type":"ContainerDied","Data":"69ef1544d51b7ab60dec70785bf924968bb314dba00d09ff78da19ec235620ca"} Nov 03 14:15:20.709241 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.708652 4924 scope.go:122] "RemoveContainer" containerID="4e33d21b89d3c9c2dc8d49594e253ad5928a9a41be3bfccf520725179f5548c9" Nov 03 14:15:20.710148 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.710120 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" Nov 03 14:15:20.710258 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.710183 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz" event={"ID":"9f707415-eaee-4191-8deb-2b21e34a4922","Type":"ContainerDied","Data":"eba6dd7138580410b417913abeb58878e688f4ac8f8ce9c67dd7af02a5c4c166"} Nov 03 14:15:20.717015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.716983 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" event={"ID":"95687e27-c3ff-4a61-8a3c-1eaeceac80e4","Type":"ContainerStarted","Data":"5c284ab50880a2b9521b216da7861674cc86cfb7f5ed7174b36e43de19b68f29"} Nov 03 14:15:20.721896 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.721875 4924 scope.go:122] "RemoveContainer" containerID="66068cc48b8f8ca4e267a4ea2aaafbc341063485b540946943cdf74e12f5085e" Nov 03 14:15:20.723237 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.723215 4924 generic.go:356] "Generic (PLEG): container finished" podID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerID="ae87c70289c802642e10b107a03732e52bc037855f23c90e89c7d39af0b334d7" exitCode=60 Nov 03 14:15:20.723351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.723256 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" event={"ID":"5d5720f5-f068-4e50-be5a-e4a2f9cae662","Type":"ContainerDied","Data":"ae87c70289c802642e10b107a03732e52bc037855f23c90e89c7d39af0b334d7"} Nov 03 14:15:20.723611 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.723596 4924 scope.go:122] "RemoveContainer" containerID="ae87c70289c802642e10b107a03732e52bc037855f23c90e89c7d39af0b334d7" Nov 03 14:15:20.723845 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:20.723828 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz_odh-ai-gateway-infra(5d5720f5-f068-4e50-be5a-e4a2f9cae662)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" Nov 03 14:15:20.736282 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.736259 4924 scope.go:122] "RemoveContainer" containerID="85e584911204439d37d78ede5116d5451d5f6f162dc9d3d935e39856838da11b" Nov 03 14:15:20.810365 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.810233 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz"] Nov 03 14:15:20.866899 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.866862 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz"] Nov 03 14:15:20.872483 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.872101 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895255-4tmtz"] Nov 03 14:15:20.896873 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.896846 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:20.909444 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.909420 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:20.951922 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:20.951893 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" path="/var/lib/kubelet/pods/9f707415-eaee-4191-8deb-2b21e34a4922/volumes" Nov 03 14:15:21.068942 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.068907 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c4285e-17f8-4972-aa11-873c8aff6f8f-kube-api-access-npzdn\" (UniqueName: \"kubernetes.io/projected/99c4285e-17f8-4972-aa11-873c8aff6f8f-kube-api-access-npzdn\") pod \"99c4285e-17f8-4972-aa11-873c8aff6f8f\" (UID: \"99c4285e-17f8-4972-aa11-873c8aff6f8f\") " Nov 03 14:15:21.069147 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.068984 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9586e6bb-4bd4-43a6-a283-955053cf2b26-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9586e6bb-4bd4-43a6-a283-955053cf2b26-maas-api-service-ca\") pod \"9586e6bb-4bd4-43a6-a283-955053cf2b26\" (UID: \"9586e6bb-4bd4-43a6-a283-955053cf2b26\") " Nov 03 14:15:21.069147 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.069131 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9586e6bb-4bd4-43a6-a283-955053cf2b26-kube-api-access-vnt2k\" (UniqueName: \"kubernetes.io/projected/9586e6bb-4bd4-43a6-a283-955053cf2b26-kube-api-access-vnt2k\") pod \"9586e6bb-4bd4-43a6-a283-955053cf2b26\" (UID: \"9586e6bb-4bd4-43a6-a283-955053cf2b26\") " Nov 03 14:15:21.069298 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.069206 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c4285e-17f8-4972-aa11-873c8aff6f8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c4285e-17f8-4972-aa11-873c8aff6f8f-maas-api-service-ca\") pod \"99c4285e-17f8-4972-aa11-873c8aff6f8f\" (UID: \"99c4285e-17f8-4972-aa11-873c8aff6f8f\") " Nov 03 14:15:21.069434 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.069386 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9586e6bb-4bd4-43a6-a283-955053cf2b26-maas-api-service-ca" pod "9586e6bb-4bd4-43a6-a283-955053cf2b26" (UID: "9586e6bb-4bd4-43a6-a283-955053cf2b26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:21.069627 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.069609 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9586e6bb-4bd4-43a6-a283-955053cf2b26-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:21.069899 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.069853 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c4285e-17f8-4972-aa11-873c8aff6f8f-maas-api-service-ca" pod "99c4285e-17f8-4972-aa11-873c8aff6f8f" (UID: "99c4285e-17f8-4972-aa11-873c8aff6f8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:21.071866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.071846 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9586e6bb-4bd4-43a6-a283-955053cf2b26-kube-api-access-vnt2k" pod "9586e6bb-4bd4-43a6-a283-955053cf2b26" (UID: "9586e6bb-4bd4-43a6-a283-955053cf2b26"). InnerVolumeSpecName "kube-api-access-vnt2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:21.071957 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.071879 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c4285e-17f8-4972-aa11-873c8aff6f8f-kube-api-access-npzdn" pod "99c4285e-17f8-4972-aa11-873c8aff6f8f" (UID: "99c4285e-17f8-4972-aa11-873c8aff6f8f"). InnerVolumeSpecName "kube-api-access-npzdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:21.076956 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.076927 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78"] Nov 03 14:15:21.080325 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.080305 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895255-hgh78"] Nov 03 14:15:21.170523 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.170482 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npzdn\" (UniqueName: \"kubernetes.io/projected/99c4285e-17f8-4972-aa11-873c8aff6f8f-kube-api-access-npzdn\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:21.170523 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.170513 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnt2k\" (UniqueName: \"kubernetes.io/projected/9586e6bb-4bd4-43a6-a283-955053cf2b26-kube-api-access-vnt2k\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:21.170523 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.170525 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c4285e-17f8-4972-aa11-873c8aff6f8f-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:21.731852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.731815 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" Nov 03 14:15:21.731852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.731843 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h" event={"ID":"9586e6bb-4bd4-43a6-a283-955053cf2b26","Type":"ContainerDied","Data":"c2c07aaca50262ac785b64e4459cc6da772008968a3e551b56428107f8e662cc"} Nov 03 14:15:21.732492 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.731910 4924 scope.go:122] "RemoveContainer" containerID="9f86fbb790bc0c14f5654b8769df34a0142fed6e38a71b3bb0fc25bbdd62fc17" Nov 03 14:15:21.734369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.734348 4924 generic.go:356] "Generic (PLEG): container finished" podID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerID="5c284ab50880a2b9521b216da7861674cc86cfb7f5ed7174b36e43de19b68f29" exitCode=60 Nov 03 14:15:21.734502 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.734375 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" event={"ID":"95687e27-c3ff-4a61-8a3c-1eaeceac80e4","Type":"ContainerDied","Data":"5c284ab50880a2b9521b216da7861674cc86cfb7f5ed7174b36e43de19b68f29"} Nov 03 14:15:21.734803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.734785 4924 scope.go:122] "RemoveContainer" containerID="5c284ab50880a2b9521b216da7861674cc86cfb7f5ed7174b36e43de19b68f29" Nov 03 14:15:21.735116 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:21.735093 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895255-pf94k_odh-ai-gateway-infra(95687e27-c3ff-4a61-8a3c-1eaeceac80e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" Nov 03 14:15:21.744776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.744757 4924 scope.go:122] "RemoveContainer" containerID="387ea723a65adbf848916793bb029806ebd67f9a0270c0e35b171bc9de997809" Nov 03 14:15:21.744976 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.744962 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" Nov 03 14:15:21.745107 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.744965 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9" event={"ID":"99c4285e-17f8-4972-aa11-873c8aff6f8f","Type":"ContainerDied","Data":"29443d8919f35328115f966e70ec2feaf30880e5f5829d23649f6e6a69c5f628"} Nov 03 14:15:21.768239 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.768209 4924 scope.go:122] "RemoveContainer" containerID="68ddaeb861001bc6faec0cdf5ef34bf21482695fe291de1af20e89d9a1d0147f" Nov 03 14:15:21.886960 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.886925 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h"] Nov 03 14:15:21.892373 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.892347 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895255-n2c7h"] Nov 03 14:15:21.912656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.912629 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:21.993848 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.993762 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9"] Nov 03 14:15:21.998911 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:21.998885 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895255-l5rs9"] Nov 03 14:15:22.081806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.081765 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5720f5-f068-4e50-be5a-e4a2f9cae662-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5720f5-f068-4e50-be5a-e4a2f9cae662-maas-api-service-ca\") pod \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\" (UID: \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\") " Nov 03 14:15:22.081960 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.081901 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5720f5-f068-4e50-be5a-e4a2f9cae662-kube-api-access-rngdx\" (UniqueName: \"kubernetes.io/projected/5d5720f5-f068-4e50-be5a-e4a2f9cae662-kube-api-access-rngdx\") pod \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\" (UID: \"5d5720f5-f068-4e50-be5a-e4a2f9cae662\") " Nov 03 14:15:22.082210 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.082185 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5720f5-f068-4e50-be5a-e4a2f9cae662-maas-api-service-ca" pod "5d5720f5-f068-4e50-be5a-e4a2f9cae662" (UID: "5d5720f5-f068-4e50-be5a-e4a2f9cae662"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:22.082383 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.082364 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5720f5-f068-4e50-be5a-e4a2f9cae662-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:22.083995 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.083974 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5720f5-f068-4e50-be5a-e4a2f9cae662-kube-api-access-rngdx" pod "5d5720f5-f068-4e50-be5a-e4a2f9cae662" (UID: "5d5720f5-f068-4e50-be5a-e4a2f9cae662"). InnerVolumeSpecName "kube-api-access-rngdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:22.183469 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.183435 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rngdx\" (UniqueName: \"kubernetes.io/projected/5d5720f5-f068-4e50-be5a-e4a2f9cae662-kube-api-access-rngdx\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:22.299581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.299493 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k"] Nov 03 14:15:22.755622 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.755592 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" Nov 03 14:15:22.756036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.755616 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz" event={"ID":"5d5720f5-f068-4e50-be5a-e4a2f9cae662","Type":"ContainerDied","Data":"14302dc9ce388a2ed428a6f4b288c11f29f95dbb9394420311fe1b9ff3624e53"} Nov 03 14:15:22.756036 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.755723 4924 scope.go:122] "RemoveContainer" containerID="ae87c70289c802642e10b107a03732e52bc037855f23c90e89c7d39af0b334d7" Nov 03 14:15:22.914371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.914342 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:22.951566 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.951525 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f0cc36-c074-42f2-9394-f864905b7187" path="/var/lib/kubelet/pods/43f0cc36-c074-42f2-9394-f864905b7187/volumes" Nov 03 14:15:22.952043 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.952022 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" path="/var/lib/kubelet/pods/9586e6bb-4bd4-43a6-a283-955053cf2b26/volumes" Nov 03 14:15:22.952483 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.952464 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" path="/var/lib/kubelet/pods/99c4285e-17f8-4972-aa11-873c8aff6f8f/volumes" Nov 03 14:15:22.968215 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.968182 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz"] Nov 03 14:15:22.972978 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.972955 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895255-tz2mz"] Nov 03 14:15:22.999550 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.999513 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-kube-api-access-bd77h\" (UniqueName: \"kubernetes.io/projected/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-kube-api-access-bd77h\") pod \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\" (UID: \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\") " Nov 03 14:15:22.999740 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:22.999571 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-maas-api-service-ca\") pod \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\" (UID: \"95687e27-c3ff-4a61-8a3c-1eaeceac80e4\") " Nov 03 14:15:23.000028 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.000001 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-maas-api-service-ca" pod "95687e27-c3ff-4a61-8a3c-1eaeceac80e4" (UID: "95687e27-c3ff-4a61-8a3c-1eaeceac80e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:23.001826 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.001804 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-kube-api-access-bd77h" pod "95687e27-c3ff-4a61-8a3c-1eaeceac80e4" (UID: "95687e27-c3ff-4a61-8a3c-1eaeceac80e4"). InnerVolumeSpecName "kube-api-access-bd77h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:23.101483 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.101449 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd77h\" (UniqueName: \"kubernetes.io/projected/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-kube-api-access-bd77h\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:23.101483 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.101480 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95687e27-c3ff-4a61-8a3c-1eaeceac80e4-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:23.765629 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.765600 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" Nov 03 14:15:23.765629 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.765614 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k" event={"ID":"95687e27-c3ff-4a61-8a3c-1eaeceac80e4","Type":"ContainerDied","Data":"b37fec4edc383393fa7021e8fbf0c225f39890e85cbfb5e7502a0fa1606cbe78"} Nov 03 14:15:23.766214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.765686 4924 scope.go:122] "RemoveContainer" containerID="5c284ab50880a2b9521b216da7861674cc86cfb7f5ed7174b36e43de19b68f29" Nov 03 14:15:23.919626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.919594 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k"] Nov 03 14:15:23.966171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:23.966140 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895255-pf94k"] Nov 03 14:15:24.951088 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:24.951038 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" path="/var/lib/kubelet/pods/5d5720f5-f068-4e50-be5a-e4a2f9cae662/volumes" Nov 03 14:15:24.951517 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:24.951500 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" path="/var/lib/kubelet/pods/95687e27-c3ff-4a61-8a3c-1eaeceac80e4/volumes" Nov 03 14:15:25.380562 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:25.380474 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:28.402139 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:28.402107 4924 scope.go:122] "RemoveContainer" containerID="efe450de8bada22c9a3a50de81e5baa654ff0ba4125e916c98c0587ee6dfb0e2" Nov 03 14:15:30.574258 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:30.574220 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:34.023590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:34.023551 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:35.668530 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:35.668490 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:40.449594 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.449499 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:40.570916 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.570879 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:40.908363 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.908330 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf"] Nov 03 14:15:40.909101 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909083 4924 cpu_manager.go:414] "containerMap: removing container" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:15:40.909160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909104 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:15:40.909160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909122 4924 cpu_manager.go:414] "containerMap: removing container" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" Nov 03 14:15:40.909160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909127 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" Nov 03 14:15:40.909160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909135 4924 cpu_manager.go:414] "containerMap: removing container" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:15:40.909160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909140 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:15:40.909160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909148 4924 cpu_manager.go:414] "containerMap: removing container" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:15:40.909160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909153 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909164 4924 cpu_manager.go:414] "containerMap: removing container" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909168 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909177 4924 cpu_manager.go:414] "containerMap: removing container" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909181 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909188 4924 cpu_manager.go:414] "containerMap: removing container" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909192 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909198 4924 cpu_manager.go:414] "containerMap: removing container" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909202 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909210 4924 cpu_manager.go:414] "containerMap: removing container" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909214 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909232 4924 cpu_manager.go:414] "containerMap: removing container" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909236 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909242 4924 cpu_manager.go:414] "containerMap: removing container" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909247 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909258 4924 cpu_manager.go:414] "containerMap: removing container" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909263 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909271 4924 cpu_manager.go:414] "containerMap: removing container" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909276 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909286 4924 cpu_manager.go:414] "containerMap: removing container" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909291 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909299 4924 cpu_manager.go:414] "containerMap: removing container" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909303 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:15:40.909375 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909380 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909389 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909396 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909401 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909407 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909412 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909418 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909424 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f707415-eaee-4191-8deb-2b21e34a4922" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909430 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909435 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909441 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909446 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909450 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909457 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909462 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:15:40.910084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.909467 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:15:40.912672 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.912657 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:40.940647 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.940618 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-metrics-certs\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:40.940794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.940670 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgw2\" (UniqueName: \"kubernetes.io/projected/56d1889a-d97a-4712-93fc-3e2e8ad2d198-kube-api-access-2rgw2\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:40.940794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.940772 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:40.940875 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:40.940806 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1889a-d97a-4712-93fc-3e2e8ad2d198-openshift-service-ca\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.009024 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.008988 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-89273860-serving-cert\"" Nov 03 14:15:41.012299 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.012275 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf"] Nov 03 14:15:41.041670 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.041636 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-metrics-certs\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.041898 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.041710 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgw2\" (UniqueName: \"kubernetes.io/projected/56d1889a-d97a-4712-93fc-3e2e8ad2d198-kube-api-access-2rgw2\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.041898 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.041751 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.041898 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.041784 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1889a-d97a-4712-93fc-3e2e8ad2d198-openshift-service-ca\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.042294 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:41.041894 4924 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-89273860-serving-cert: secret "maas-api-e2e-mt-89273860-serving-cert" not found Nov 03 14:15:41.042294 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:41.042163 4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls podName:56d1889a-d97a-4712-93fc-3e2e8ad2d198 nodeName:}" failed. No retries permitted until 2026-11-03 14:15:41.54213766 +0000 UTC m=+4274.736541254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls") pod "maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" (UID: "56d1889a-d97a-4712-93fc-3e2e8ad2d198") : secret "maas-api-e2e-mt-89273860-serving-cert" not found Nov 03 14:15:41.043329 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.043298 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1889a-d97a-4712-93fc-3e2e8ad2d198-openshift-service-ca\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.044735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.044705 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-metrics-certs\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.088505 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.088463 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:41.096294 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.096268 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgw2\" (UniqueName: \"kubernetes.io/projected/56d1889a-d97a-4712-93fc-3e2e8ad2d198-kube-api-access-2rgw2\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.243219 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.243107 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:41.371592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.371519 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:41.494462 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.494377 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:41.548480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.548442 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.550897 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.550874 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls\") pod \"maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:41.786545 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.786456 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:41.826431 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:41.826392 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:42.198010 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:42.197976 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf"] Nov 03 14:15:42.208978 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:15:42.207940 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d1889a_d97a_4712_93fc_3e2e8ad2d198.slice/crio-023e6f77cd4d519098ed79a3c86d962fd281789026736a2a073e429fed0b461d WatchSource:0}: Error finding container 023e6f77cd4d519098ed79a3c86d962fd281789026736a2a073e429fed0b461d: Status 404 returned error can't find the container with id 023e6f77cd4d519098ed79a3c86d962fd281789026736a2a073e429fed0b461d Nov 03 14:15:42.209670 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:42.209648 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:15:42.399093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:42.398776 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:42.540049 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:42.539968 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:42.909256 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:42.909222 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" event={"ID":"56d1889a-d97a-4712-93fc-3e2e8ad2d198","Type":"ContainerStarted","Data":"023e6f77cd4d519098ed79a3c86d962fd281789026736a2a073e429fed0b461d"} Nov 03 14:15:43.917410 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:43.917369 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" event={"ID":"56d1889a-d97a-4712-93fc-3e2e8ad2d198","Type":"ContainerStarted","Data":"b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23"} Nov 03 14:15:43.917865 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:43.917439 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:44.028971 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:44.028919 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" podStartSLOduration=3.072432344 podStartE2EDuration="4.028905206s" totalImagesPullingTime="956.472862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:44.025669301 +0000 UTC m=+4277.220072895" watchObservedRunningTime="2026-11-03 14:15:44.028905206 +0000 UTC m=+4277.223308799" Nov 03 14:15:46.659287 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:46.659246 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:46.930363 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:46.930271 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:47.154516 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:47.154485 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:47.308378 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:47.308256 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:48.561063 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:15:48.561024 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:15:49.929949 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:49.929920 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:15:50.233965 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:50.233878 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:50.418959 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:50.418610 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:52.215530 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:52.215495 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:15:58.009975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:15:58.009940 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:03.545846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:03.545811 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:08.920563 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:08.920468 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:13.956680 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:13.956641 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:18.560252 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:16:18.560210 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:16:19.088678 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:19.088640 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:24.192878 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:24.192843 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:29.310783 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:29.310742 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:34.437246 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:34.437205 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:39.500829 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:39.500783 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:40.823192 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:40.823149 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:44.578580 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:44.578540 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:48.558388 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:16:48.558349 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:16:49.682872 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:49.682832 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:54.922977 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:54.922939 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:55.462512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:55.462454 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:58.148524 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:58.148490 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:16:59.967301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:16:59.967264 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:05.070807 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:05.070765 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:07.069579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.069499 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn"] Nov 03 14:17:07.070056 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070041 4924 cpu_manager.go:414] "containerMap: removing container" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070057 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c4285e-17f8-4972-aa11-873c8aff6f8f" containerName="cleanup" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070084 4924 cpu_manager.go:414] "containerMap: removing container" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070090 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081b76f0-1d17-41ed-b0ba-9256d5bd919c" containerName="istio-proxy" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070099 4924 cpu_manager.go:414] "containerMap: removing container" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070104 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070117 4924 cpu_manager.go:414] "containerMap: removing container" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070122 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070130 4924 cpu_manager.go:414] "containerMap: removing container" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:17:07.070145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070136 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:17:07.070430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070213 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9586e6bb-4bd4-43a6-a283-955053cf2b26" containerName="cleanup" Nov 03 14:17:07.070430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070225 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="95687e27-c3ff-4a61-8a3c-1eaeceac80e4" containerName="cleanup" Nov 03 14:17:07.070430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070234 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f0cc36-c074-42f2-9394-f864905b7187" containerName="cleanup" Nov 03 14:17:07.070430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.070240 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5720f5-f068-4e50-be5a-e4a2f9cae662" containerName="cleanup" Nov 03 14:17:07.073390 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.073370 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.088295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.088270 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-home\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.088432 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.088303 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-dshm\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.088507 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.088475 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sj9w\" (UniqueName: \"kubernetes.io/projected/42ead058-3e92-4458-9529-596b2c49e135-kube-api-access-2sj9w\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.088606 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.088590 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-model-cache\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.088686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.088671 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-tmp-dir\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.088771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.088751 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ead058-3e92-4458-9529-596b2c49e135-tls-certs\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.126357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.124842 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"kube-root-ca.crt\"" Nov 03 14:17:07.144528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.144500 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"openshift-service-ca.crt\"" Nov 03 14:17:07.144528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.144500 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"default-dockercfg-4n97f\"" Nov 03 14:17:07.171516 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.171486 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"rate-test-model-60261a-kserve-self-signed-certs\"" Nov 03 14:17:07.172357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.172332 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:07.176701 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.176681 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn"] Nov 03 14:17:07.189596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.189566 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sj9w\" (UniqueName: \"kubernetes.io/projected/42ead058-3e92-4458-9529-596b2c49e135-kube-api-access-2sj9w\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.189707 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.189626 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-model-cache\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.189707 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.189674 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-tmp-dir\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.189707 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.189692 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ead058-3e92-4458-9529-596b2c49e135-tls-certs\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.189862 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.189716 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-home\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.189862 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.189845 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-dshm\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.190170 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.190147 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-tmp-dir\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.190249 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.190165 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-model-cache\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.190249 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.190190 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-home\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.192029 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.192002 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-dshm\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.192318 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.192301 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ead058-3e92-4458-9529-596b2c49e135-tls-certs\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.245235 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.245197 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sj9w\" (UniqueName: \"kubernetes.io/projected/42ead058-3e92-4458-9529-596b2c49e135-kube-api-access-2sj9w\") pod \"rate-test-model-60261a-kserve-65995c99bf-tzrbn\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.386327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.386233 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:07.763052 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:07.763022 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn"] Nov 03 14:17:07.763904 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:17:07.763882 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42ead058_3e92_4458_9529_596b2c49e135.slice/crio-2716befe8ca8d0b076610f933e86adc049858a987c4852134cc00dcdbfb71f13 WatchSource:0}: Error finding container 2716befe8ca8d0b076610f933e86adc049858a987c4852134cc00dcdbfb71f13: Status 404 returned error can't find the container with id 2716befe8ca8d0b076610f933e86adc049858a987c4852134cc00dcdbfb71f13 Nov 03 14:17:08.503674 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:08.503631 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" event={"ID":"42ead058-3e92-4458-9529-596b2c49e135","Type":"ContainerStarted","Data":"da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9"} Nov 03 14:17:08.504114 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:08.503685 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" event={"ID":"42ead058-3e92-4458-9529-596b2c49e135","Type":"ContainerStarted","Data":"2716befe8ca8d0b076610f933e86adc049858a987c4852134cc00dcdbfb71f13"} Nov 03 14:17:08.613402 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:08.613347 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" podStartSLOduration=1.613332545 podStartE2EDuration="1.613332545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:08.610577736 +0000 UTC m=+4361.804981330" watchObservedRunningTime="2026-11-03 14:17:08.613332545 +0000 UTC m=+4361.807736138" Nov 03 14:17:09.386960 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:09.386923 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:10.207295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:10.207262 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:10.370409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:10.370365 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:10.491819 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:10.491726 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:14.830308 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:14.830250 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:15.668652 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:15.668608 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:15.818755 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:15.818681 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:15.954554 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:15.954430 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:16.268008 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:16.267898 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:17.387148 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:17.387115 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:17.400563 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:17.400536 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:17.574790 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:17.574756 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:17:18.563089 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:17:18.563037 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:17:20.970039 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:20.970002 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:22.221743 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:22.221706 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:22.498495 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:22.498391 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:22.642669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:22.642622 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:23.257629 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:23.257593 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:26.064695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:26.064659 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:28.465871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:28.465832 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:28.879996 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:28.879911 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:29.006867 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:29.006827 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:29.969571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:29.969480 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:31.644640 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:31.644567 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:36.210719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:36.210687 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:37.068025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:37.067984 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:41.157378 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:41.157338 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:41.280200 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:41.280159 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:42.277334 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:42.277291 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:47.385871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:47.385834 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:48.558686 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:17:48.558649 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:17:52.458476 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:52.458429 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:17:57.468051 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:17:57.468009 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:02.493862 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:02.493817 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:07.587977 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:07.587937 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:12.811853 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:12.811778 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:15.804232 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.804199 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw"] Nov 03 14:18:15.807338 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.807318 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:15.876340 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.876304 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:15.904185 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.904145 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"kube-root-ca.crt\"" Nov 03 14:18:15.904389 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.904155 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"openshift-service-ca.crt\"" Nov 03 14:18:15.904389 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.904214 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"rate-test-model-367381-kserve-self-signed-certs\"" Nov 03 14:18:15.904389 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.904318 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"default-dockercfg-4g28d\"" Nov 03 14:18:15.907760 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.907735 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw"] Nov 03 14:18:15.943956 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.943927 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zvl\" (UniqueName: \"kubernetes.io/projected/ecc6c0f9-ab3c-45e2-9552-e956521e6754-kube-api-access-m2zvl\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:15.944129 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.943974 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-home\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:15.944129 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.944061 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-dshm\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:15.944226 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.944154 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tls-certs\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:15.944226 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.944195 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tmp-dir\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:15.944314 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:15.944245 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-model-cache\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045032 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tls-certs\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045269 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045105 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tmp-dir\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045269 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045170 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-model-cache\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045269 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045241 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zvl\" (UniqueName: \"kubernetes.io/projected/ecc6c0f9-ab3c-45e2-9552-e956521e6754-kube-api-access-m2zvl\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045451 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045293 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-home\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045451 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045345 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-dshm\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045573 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045509 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tmp-dir\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045705 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045684 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-model-cache\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.045790 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.045765 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-home\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.047621 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.047601 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tls-certs\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.047621 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.047613 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-dshm\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.103024 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.102950 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zvl\" (UniqueName: \"kubernetes.io/projected/ecc6c0f9-ab3c-45e2-9552-e956521e6754-kube-api-access-m2zvl\") pod \"rate-test-model-367381-kserve-55946c587c-pqqlw\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.118286 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.118244 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:16.497261 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:18:16.497227 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecc6c0f9_ab3c_45e2_9552_e956521e6754.slice/crio-34dd0ff8a4c6e0f15ff2035bb016d1907937370b9d29ef42c8a039e2e01ce106 WatchSource:0}: Error finding container 34dd0ff8a4c6e0f15ff2035bb016d1907937370b9d29ef42c8a039e2e01ce106: Status 404 returned error can't find the container with id 34dd0ff8a4c6e0f15ff2035bb016d1907937370b9d29ef42c8a039e2e01ce106 Nov 03 14:18:16.497764 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.497744 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw"] Nov 03 14:18:16.988930 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.988891 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" event={"ID":"ecc6c0f9-ab3c-45e2-9552-e956521e6754","Type":"ContainerStarted","Data":"ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e"} Nov 03 14:18:16.989430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:16.988939 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" event={"ID":"ecc6c0f9-ab3c-45e2-9552-e956521e6754","Type":"ContainerStarted","Data":"34dd0ff8a4c6e0f15ff2035bb016d1907937370b9d29ef42c8a039e2e01ce106"} Nov 03 14:18:17.022252 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:17.022212 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:17.106178 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:17.106112 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" podStartSLOduration=2.106093442 podStartE2EDuration="2.106093442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:18:17.101470298 +0000 UTC m=+4430.295873893" watchObservedRunningTime="2026-11-03 14:18:17.106093442 +0000 UTC m=+4430.300497038" Nov 03 14:18:17.118672 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:17.118640 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:17.149477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:17.149438 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:17.811528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:17.811467 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:18.560100 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:18:18.560039 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:18:22.794797 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:22.794764 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:26.118743 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:26.118712 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:26.131260 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:26.131233 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:27.074001 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:27.073968 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:18:27.873890 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:27.873843 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:29.054323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:29.054282 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7"] Nov 03 14:18:29.099125 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:29.099090 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61bc07e9-z55m7"] Nov 03 14:18:30.951286 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:30.951250 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a8e841-252f-4396-94cd-32536e18f32c" path="/var/lib/kubelet/pods/d9a8e841-252f-4396-94cd-32536e18f32c/volumes" Nov 03 14:18:32.274690 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.274646 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf"] Nov 03 14:18:32.275172 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.274913 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" podUID="56d1889a-d97a-4712-93fc-3e2e8ad2d198" containerName="maas-api" containerID="cri-o://b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23" gracePeriod=30 Nov 03 14:18:32.359878 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.359848 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:32.594213 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.594168 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:18:32.662030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.661996 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d1889a-d97a-4712-93fc-3e2e8ad2d198-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1889a-d97a-4712-93fc-3e2e8ad2d198-openshift-service-ca\") pod \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " Nov 03 14:18:32.662409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.662384 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-metrics-certs\") pod \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " Nov 03 14:18:32.662643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.662623 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls\") pod \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " Nov 03 14:18:32.662806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.662789 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d1889a-d97a-4712-93fc-3e2e8ad2d198-kube-api-access-2rgw2\" (UniqueName: \"kubernetes.io/projected/56d1889a-d97a-4712-93fc-3e2e8ad2d198-kube-api-access-2rgw2\") pod \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\" (UID: \"56d1889a-d97a-4712-93fc-3e2e8ad2d198\") " Nov 03 14:18:32.663205 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.662640 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d1889a-d97a-4712-93fc-3e2e8ad2d198-openshift-service-ca" pod "56d1889a-d97a-4712-93fc-3e2e8ad2d198" (UID: "56d1889a-d97a-4712-93fc-3e2e8ad2d198"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:18:32.671139 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.665903 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls" pod "56d1889a-d97a-4712-93fc-3e2e8ad2d198" (UID: "56d1889a-d97a-4712-93fc-3e2e8ad2d198"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:18:32.676121 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.671654 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d1889a-d97a-4712-93fc-3e2e8ad2d198-kube-api-access-2rgw2" pod "56d1889a-d97a-4712-93fc-3e2e8ad2d198" (UID: "56d1889a-d97a-4712-93fc-3e2e8ad2d198"). InnerVolumeSpecName "kube-api-access-2rgw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:18:32.715278 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.715238 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-metrics-certs" pod "56d1889a-d97a-4712-93fc-3e2e8ad2d198" (UID: "56d1889a-d97a-4712-93fc-3e2e8ad2d198"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:18:32.764284 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.764241 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgw2\" (UniqueName: \"kubernetes.io/projected/56d1889a-d97a-4712-93fc-3e2e8ad2d198-kube-api-access-2rgw2\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:32.764284 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.764276 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d1889a-d97a-4712-93fc-3e2e8ad2d198-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:32.764284 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.764293 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:32.764553 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:32.764307 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d1889a-d97a-4712-93fc-3e2e8ad2d198-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:33.105989 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.105948 4924 generic.go:356] "Generic (PLEG): container finished" podID="56d1889a-d97a-4712-93fc-3e2e8ad2d198" containerID="b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23" exitCode=0 Nov 03 14:18:33.106169 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.105993 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" event={"ID":"56d1889a-d97a-4712-93fc-3e2e8ad2d198","Type":"ContainerDied","Data":"b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23"} Nov 03 14:18:33.106169 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.106051 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" event={"ID":"56d1889a-d97a-4712-93fc-3e2e8ad2d198","Type":"ContainerDied","Data":"023e6f77cd4d519098ed79a3c86d962fd281789026736a2a073e429fed0b461d"} Nov 03 14:18:33.106169 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.106004 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf" Nov 03 14:18:33.106169 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.106084 4924 scope.go:122] "RemoveContainer" containerID="b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23" Nov 03 14:18:33.116473 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.116453 4924 scope.go:122] "RemoveContainer" containerID="b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23" Nov 03 14:18:33.116737 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:18:33.116716 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23\": container with ID starting with b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23 not found: ID does not exist" containerID="b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23" Nov 03 14:18:33.116779 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.116746 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23"} err="failed to get container status \"b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23\": rpc error: code = NotFound desc = could not find container \"b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23\": container with ID starting with b625d641f3a0ed60f93fa20613fdfa0249bb0a69ddd7731ee91d2c9323994a23 not found: ID does not exist" Nov 03 14:18:33.306311 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.306135 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf"] Nov 03 14:18:33.306311 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.306200 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89273860-6fd6cf5bd-xmkpf"] Nov 03 14:18:33.947658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:33.947598 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:34.080525 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:34.080489 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:34.273904 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:34.273818 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:34.953231 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:34.953187 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d1889a-d97a-4712-93fc-3e2e8ad2d198" path="/var/lib/kubelet/pods/56d1889a-d97a-4712-93fc-3e2e8ad2d198/volumes" Nov 03 14:18:35.551799 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:35.551765 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:36.159093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:36.158526 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:36.615854 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:36.615764 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:36.755923 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:36.755886 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:37.257551 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:37.257517 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:39.794005 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:39.793955 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:41.482541 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:41.482506 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:42.930336 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:42.930299 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:43.054720 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:43.054677 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:43.182110 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:43.181983 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:45.277225 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:45.277191 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:45.396425 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:45.396369 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:48.207385 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:48.207351 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:48.558762 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:18:48.558682 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:18:50.543527 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:50.543496 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:52.756153 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:52.756115 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:52.883151 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:52.883109 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:53.112173 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:53.112097 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:54.352400 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:54.352362 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:54.479840 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:54.479803 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:55.664622 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:55.664583 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:18:59.899655 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:59.899619 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp"] Nov 03 14:18:59.900209 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:59.900192 4924 cpu_manager.go:414] "containerMap: removing container" podUID="56d1889a-d97a-4712-93fc-3e2e8ad2d198" containerName="maas-api" Nov 03 14:18:59.900271 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:59.900210 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d1889a-d97a-4712-93fc-3e2e8ad2d198" containerName="maas-api" Nov 03 14:18:59.900319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:59.900310 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d1889a-d97a-4712-93fc-3e2e8ad2d198" containerName="maas-api" Nov 03 14:18:59.903983 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:59.903966 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:18:59.958235 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:59.958200 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp"] Nov 03 14:18:59.987273 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:59.987232 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:18:59.999600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:18:59.999563 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:19:00.024783 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.024738 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmfr\" (UniqueName: \"kubernetes.io/projected/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-kube-api-access-7kmfr\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.024986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.024888 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.025450 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.025419 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.126958 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.126924 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmfr\" (UniqueName: \"kubernetes.io/projected/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-kube-api-access-7kmfr\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.127178 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.126969 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.127178 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.127036 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.127772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.127750 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.129507 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.129477 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.180647 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.180617 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmfr\" (UniqueName: \"kubernetes.io/projected/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-kube-api-access-7kmfr\") pod \"maas-api-revoke-sub-fd13b3e3-25gsp\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.217282 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.217244 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:00.471428 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.471331 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:00.590158 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.590124 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp"] Nov 03 14:19:00.591884 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:19:00.591854 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0b8e4d_5ce0_48a3_82cd_b8db20efd8fc.slice/crio-e2e3e76ad391f3f1c009c9a4022a741cb1af8553cde188743cab8459ff237473 WatchSource:0}: Error finding container e2e3e76ad391f3f1c009c9a4022a741cb1af8553cde188743cab8459ff237473: Status 404 returned error can't find the container with id e2e3e76ad391f3f1c009c9a4022a741cb1af8553cde188743cab8459ff237473 Nov 03 14:19:00.776968 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:00.776871 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:01.312037 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:01.311946 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" event={"ID":"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc","Type":"ContainerStarted","Data":"acac23ac85117de202ad9e9b06238d7107fce2feedfd8abb3c44be2886c93ea1"} Nov 03 14:19:01.312037 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:01.312007 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" event={"ID":"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc","Type":"ContainerStarted","Data":"e2e3e76ad391f3f1c009c9a4022a741cb1af8553cde188743cab8459ff237473"} Nov 03 14:19:01.424399 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:01.424338 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" podStartSLOduration=2.424318463 podStartE2EDuration="2.424318463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:01.418866026 +0000 UTC m=+4474.613269624" watchObservedRunningTime="2026-11-03 14:19:01.424318463 +0000 UTC m=+4474.618722058" Nov 03 14:19:01.478688 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:01.478647 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:01.598165 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:01.598008 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:02.320291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:02.320253 4924 generic.go:356] "Generic (PLEG): container finished" podID="5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc" containerID="acac23ac85117de202ad9e9b06238d7107fce2feedfd8abb3c44be2886c93ea1" exitCode=0 Nov 03 14:19:02.320711 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:02.320333 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" event={"ID":"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc","Type":"ContainerDied","Data":"acac23ac85117de202ad9e9b06238d7107fce2feedfd8abb3c44be2886c93ea1"} Nov 03 14:19:03.475440 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.475413 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:03.587250 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.587166 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-kube-api-access-7kmfr\" (UniqueName: \"kubernetes.io/projected/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-kube-api-access-7kmfr\") pod \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " Nov 03 14:19:03.587250 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.587216 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-xks-ca\") pod \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " Nov 03 14:19:03.587714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.587268 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-openshift-ca\") pod \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\" (UID: \"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc\") " Nov 03 14:19:03.588119 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.588066 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-openshift-ca" pod "5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc" (UID: "5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:19:03.590522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.590500 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-kube-api-access-7kmfr" pod "5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc" (UID: "5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc"). InnerVolumeSpecName "kube-api-access-7kmfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:03.590616 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.590514 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-xks-ca" pod "5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc" (UID: "5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:03.688414 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.688376 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kmfr\" (UniqueName: \"kubernetes.io/projected/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-kube-api-access-7kmfr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.688414 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.688414 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.688620 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:03.688424 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:04.048622 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:04.048582 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9"] Nov 03 14:19:04.053428 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:04.053395 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8590ea-kklz9"] Nov 03 14:19:04.335540 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:04.335463 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" Nov 03 14:19:04.335540 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:04.335500 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp" event={"ID":"5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc","Type":"ContainerDied","Data":"e2e3e76ad391f3f1c009c9a4022a741cb1af8553cde188743cab8459ff237473"} Nov 03 14:19:04.335735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:04.335550 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e3e76ad391f3f1c009c9a4022a741cb1af8553cde188743cab8459ff237473" Nov 03 14:19:04.951768 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:04.951736 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae" path="/var/lib/kubelet/pods/d867b821-2ea6-4fbf-b7a1-b9e3cf3ebbae/volumes" Nov 03 14:19:06.076337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:06.076285 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:06.265115 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:06.265079 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:07.144350 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:07.144318 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:11.292261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:11.292217 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:12.009030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:12.008993 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:12.956401 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:12.956365 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:15.556345 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:15.556301 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:16.330630 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:16.330591 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:18.560872 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:19:18.560832 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:19:18.788057 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.788021 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7"] Nov 03 14:19:18.788578 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.788562 4924 cpu_manager.go:414] "containerMap: removing container" podUID="5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc" containerName="revoke-keys" Nov 03 14:19:18.788633 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.788579 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc" containerName="revoke-keys" Nov 03 14:19:18.788684 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.788675 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc" containerName="revoke-keys" Nov 03 14:19:18.792127 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.792108 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:18.864848 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.864676 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:18.887901 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.887862 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b63d92-a712-4b7d-b87f-e3429c67107b-tls-certs\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:18.888303 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.888275 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgdf\" (UniqueName: \"kubernetes.io/projected/27b63d92-a712-4b7d-b87f-e3429c67107b-kube-api-access-2wgdf\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:18.888554 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.888512 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-tmp-dir\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:18.888724 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.888565 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-dshm\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:18.888724 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.888679 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-home\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:18.888914 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.888754 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-model-cache\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:18.891399 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.891349 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2399b8\"/\"kube-root-ca.crt\"" Nov 03 14:19:18.891571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.891472 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2399b8\"/\"test-model-2399b8-kserve-self-signed-certs\"" Nov 03 14:19:18.892242 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.892218 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2399b8\"/\"openshift-service-ca.crt\"" Nov 03 14:19:18.892387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.892313 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2399b8\"/\"default-dockercfg-9g99t\"" Nov 03 14:19:18.898416 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:18.898357 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7"] Nov 03 14:19:19.001094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.000909 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgdf\" (UniqueName: \"kubernetes.io/projected/27b63d92-a712-4b7d-b87f-e3429c67107b-kube-api-access-2wgdf\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.006090 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.001325 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-tmp-dir\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.006090 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.001762 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-tmp-dir\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.025111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.023775 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-dshm\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.025111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.023840 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-home\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.025111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.023880 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-model-cache\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.025111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.023943 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b63d92-a712-4b7d-b87f-e3429c67107b-tls-certs\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.025111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.024738 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-home\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.025553 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.025265 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-model-cache\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.052344 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.052264 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b63d92-a712-4b7d-b87f-e3429c67107b-tls-certs\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.053208 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.053184 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-dshm\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.059791 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.059763 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgdf\" (UniqueName: \"kubernetes.io/projected/27b63d92-a712-4b7d-b87f-e3429c67107b-kube-api-access-2wgdf\") pod \"test-model-2399b8-kserve-847b54869c-bpqx7\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.116329 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.116237 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:19.598593 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:19:19.598553 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27b63d92_a712_4b7d_b87f_e3429c67107b.slice/crio-166b5237c33ad2124d05c50694c847c4f9227f7230b3dd83d4a67fe65bae4ed3 WatchSource:0}: Error finding container 166b5237c33ad2124d05c50694c847c4f9227f7230b3dd83d4a67fe65bae4ed3: Status 404 returned error can't find the container with id 166b5237c33ad2124d05c50694c847c4f9227f7230b3dd83d4a67fe65bae4ed3 Nov 03 14:19:19.601804 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:19.601772 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7"] Nov 03 14:19:20.541763 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:20.541700 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" event={"ID":"27b63d92-a712-4b7d-b87f-e3429c67107b","Type":"ContainerStarted","Data":"52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b"} Nov 03 14:19:20.542091 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:20.542043 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" event={"ID":"27b63d92-a712-4b7d-b87f-e3429c67107b","Type":"ContainerStarted","Data":"166b5237c33ad2124d05c50694c847c4f9227f7230b3dd83d4a67fe65bae4ed3"} Nov 03 14:19:20.692101 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:20.690890 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" podStartSLOduration=2.6908570000000003 podStartE2EDuration="2.690857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:20.688545567 +0000 UTC m=+4493.882949162" watchObservedRunningTime="2026-11-03 14:19:20.690857 +0000 UTC m=+4493.885260595" Nov 03 14:19:21.122125 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:21.121552 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:21.570095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:21.567483 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" podUID="42ead058-3e92-4458-9529-596b2c49e135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:19:21.593133 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:21.587675 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:22.033095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:22.032724 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 14:19:23.713688 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:23.713649 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:24.583732 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:24.583697 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:24.704687 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:24.704649 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:25.761687 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:25.761656 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:25.886252 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:25.886214 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:26.366833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:26.366799 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:26.783894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:26.783862 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:27.712562 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:27.712532 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:19:27.727933 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:27.727907 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:19:28.492466 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:28.492432 4924 scope.go:122] "RemoveContainer" containerID="1a8b044808468dc236d3c3f79f60e4f16030715bbad33afe22843cf4438bba98" Nov 03 14:19:29.117465 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:29.117429 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:29.130802 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:29.130773 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:29.662112 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:29.662063 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:19:31.816587 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:31.816547 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:32.127345 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:32.127261 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:36.231346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:36.231280 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:38.004081 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:38.004045 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:41.824443 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:41.824402 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:43.063056 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:43.063003 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:45.144472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:45.144428 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:45.270583 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:45.270549 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:46.794399 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:46.794357 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:47.374027 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:47.373971 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:48.556037 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:48.556007 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:48.560326 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:19:48.560290 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:19:53.962571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:53.962531 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:55.487579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:55.487529 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:55.611738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:55.611672 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:19:58.923147 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:19:58.923088 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:04.001676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:04.001629 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:09.076669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:09.076574 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:14.091928 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:14.091874 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:18.558449 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:20:18.558407 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:20:19.181781 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:19.181751 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:24.255316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:24.255277 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:28.527138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:28.527108 4924 scope.go:122] "RemoveContainer" containerID="9dc6071890d710aa10deb75c3dcfa75e746c1a1f55c4dacde8642d08b5669423" Nov 03 14:20:29.296895 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:29.296857 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:34.370827 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:34.370792 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:39.575639 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:39.575606 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:42.161225 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:42.161177 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:42.278919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:42.278872 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:44.650197 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:44.650156 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:48.560557 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:20:48.560519 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:20:48.804330 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:48.804297 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:49.654044 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:49.654002 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:54.668317 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:54.668281 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:54.805143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:54.805039 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:58.553921 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:58.553882 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:20:59.870065 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:20:59.870021 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:05.051387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:05.051347 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:10.094623 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:10.094581 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:13.401525 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:13.401476 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:21:15.298102 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:15.298041 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:18.560445 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:21:18.560400 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:21:20.365383 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:20.365326 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:25.396891 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:25.396854 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:30.453882 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:30.453847 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:32.102400 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:32.102367 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:32.219797 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:32.219756 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:35.532514 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:35.532474 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:40.767758 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:40.767678 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:42.483099 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:42.483048 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:45.893809 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:45.893778 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:48.558970 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:21:48.558928 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:21:50.978322 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:50.978273 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:21:56.088296 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:21:56.088251 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:01.182879 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:01.182842 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:06.269176 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:06.269134 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:11.365514 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:11.365474 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:16.466586 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:16.466536 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:18.558313 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:22:18.558274 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:22:19.648902 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:19.648864 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:21.560544 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:21.560506 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:25.274199 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:25.274162 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:25.410337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:25.410307 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:26.209100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:26.209034 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:26.840468 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:26.840428 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:32.164773 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:32.164732 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:32.350416 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:32.350382 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn"] Nov 03 14:22:32.351643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:32.351609 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:32.533939 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:32.533832 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:35.210328 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:35.210291 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:35.213163 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:35.213133 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw"] Nov 03 14:22:37.292581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:37.292546 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:42.379282 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:42.379241 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:42.821656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:42.821618 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:47.373393 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.373317 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" podUID="42ead058-3e92-4458-9529-596b2c49e135" containerName="main" containerID="cri-o://da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9" gracePeriod=45 Nov 03 14:22:47.453863 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.453826 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:47.564535 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.564491 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" podUID="42ead058-3e92-4458-9529-596b2c49e135" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": dial tcp 10.132.0.177:8000: connect: connection refused" Nov 03 14:22:47.707365 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.707334 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:22:47.771611 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.771567 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-home\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-home\") pod \"42ead058-3e92-4458-9529-596b2c49e135\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " Nov 03 14:22:47.771809 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.771649 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42ead058-3e92-4458-9529-596b2c49e135-tls-certs\" (UniqueName: \"kubernetes.io/secret/42ead058-3e92-4458-9529-596b2c49e135-tls-certs\") pod \"42ead058-3e92-4458-9529-596b2c49e135\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " Nov 03 14:22:47.771809 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.771751 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ead058-3e92-4458-9529-596b2c49e135-kube-api-access-2sj9w\" (UniqueName: \"kubernetes.io/projected/42ead058-3e92-4458-9529-596b2c49e135-kube-api-access-2sj9w\") pod \"42ead058-3e92-4458-9529-596b2c49e135\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " Nov 03 14:22:47.771932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.771830 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-tmp-dir\") pod \"42ead058-3e92-4458-9529-596b2c49e135\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " Nov 03 14:22:47.771932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.771832 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-home" pod "42ead058-3e92-4458-9529-596b2c49e135" (UID: "42ead058-3e92-4458-9529-596b2c49e135"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:22:47.771932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.771894 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-model-cache\") pod \"42ead058-3e92-4458-9529-596b2c49e135\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " Nov 03 14:22:47.772144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.771929 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-dshm\") pod \"42ead058-3e92-4458-9529-596b2c49e135\" (UID: \"42ead058-3e92-4458-9529-596b2c49e135\") " Nov 03 14:22:47.772144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.772127 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-tmp-dir" pod "42ead058-3e92-4458-9529-596b2c49e135" (UID: "42ead058-3e92-4458-9529-596b2c49e135"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:22:47.772238 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.772136 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-model-cache" pod "42ead058-3e92-4458-9529-596b2c49e135" (UID: "42ead058-3e92-4458-9529-596b2c49e135"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:22:47.772440 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.772416 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:47.772570 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.772442 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:47.772570 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.772456 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:47.773928 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.773906 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-dshm" pod "42ead058-3e92-4458-9529-596b2c49e135" (UID: "42ead058-3e92-4458-9529-596b2c49e135"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:22:47.774339 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.774312 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ead058-3e92-4458-9529-596b2c49e135-tls-certs" pod "42ead058-3e92-4458-9529-596b2c49e135" (UID: "42ead058-3e92-4458-9529-596b2c49e135"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:22:47.774339 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.774317 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ead058-3e92-4458-9529-596b2c49e135-kube-api-access-2sj9w" pod "42ead058-3e92-4458-9529-596b2c49e135" (UID: "42ead058-3e92-4458-9529-596b2c49e135"). InnerVolumeSpecName "kube-api-access-2sj9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:22:47.874009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.873972 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ead058-3e92-4458-9529-596b2c49e135-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:47.874009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.874004 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sj9w\" (UniqueName: \"kubernetes.io/projected/42ead058-3e92-4458-9529-596b2c49e135-kube-api-access-2sj9w\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:47.874009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:47.874014 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ead058-3e92-4458-9529-596b2c49e135-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:48.116143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.114585 4924 generic.go:356] "Generic (PLEG): container finished" podID="42ead058-3e92-4458-9529-596b2c49e135" containerID="da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9" exitCode=0 Nov 03 14:22:48.116143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.114777 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" event={"ID":"42ead058-3e92-4458-9529-596b2c49e135","Type":"ContainerDied","Data":"da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9"} Nov 03 14:22:48.116143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.114815 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" event={"ID":"42ead058-3e92-4458-9529-596b2c49e135","Type":"ContainerDied","Data":"2716befe8ca8d0b076610f933e86adc049858a987c4852134cc00dcdbfb71f13"} Nov 03 14:22:48.116143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.114838 4924 scope.go:122] "RemoveContainer" containerID="da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9" Nov 03 14:22:48.116143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.114999 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn" Nov 03 14:22:48.138754 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.138726 4924 scope.go:122] "RemoveContainer" containerID="da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9" Nov 03 14:22:48.139115 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:22:48.139091 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9\": container with ID starting with da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9 not found: ID does not exist" containerID="da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9" Nov 03 14:22:48.139206 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.139132 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9"} err="failed to get container status \"da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9\": rpc error: code = NotFound desc = could not find container \"da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9\": container with ID starting with da89e23f44075f71028bae147217870ca92bca38be8e9aa2a2cb4414c14687a9 not found: ID does not exist" Nov 03 14:22:48.274240 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.274197 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn"] Nov 03 14:22:48.279932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.279904 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/rate-test-model-60261a-kserve-65995c99bf-tzrbn"] Nov 03 14:22:48.558090 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:22:48.558039 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:22:48.951405 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:48.951372 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ead058-3e92-4458-9529-596b2c49e135" path="/var/lib/kubelet/pods/42ead058-3e92-4458-9529-596b2c49e135/volumes" Nov 03 14:22:50.233553 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.233516 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" podUID="ecc6c0f9-ab3c-45e2-9552-e956521e6754" containerName="main" containerID="cri-o://ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e" gracePeriod=45 Nov 03 14:22:50.519576 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.519550 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:22:50.522702 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.522674 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc6c0f9-ab3c-45e2-9552-e956521e6754-kube-api-access-m2zvl\" (UniqueName: \"kubernetes.io/projected/ecc6c0f9-ab3c-45e2-9552-e956521e6754-kube-api-access-m2zvl\") pod \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " Nov 03 14:22:50.522794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.522717 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-home\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-home\") pod \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " Nov 03 14:22:50.522794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.522740 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tmp-dir\") pod \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " Nov 03 14:22:50.522794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.522770 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-model-cache\") pod \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " Nov 03 14:22:50.522922 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.522895 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-dshm\") pod \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " Nov 03 14:22:50.522970 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.522946 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-home" pod "ecc6c0f9-ab3c-45e2-9552-e956521e6754" (UID: "ecc6c0f9-ab3c-45e2-9552-e956521e6754"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:22:50.523064 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.523041 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tmp-dir" pod "ecc6c0f9-ab3c-45e2-9552-e956521e6754" (UID: "ecc6c0f9-ab3c-45e2-9552-e956521e6754"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:22:50.523145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.523093 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-model-cache" pod "ecc6c0f9-ab3c-45e2-9552-e956521e6754" (UID: "ecc6c0f9-ab3c-45e2-9552-e956521e6754"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:22:50.523308 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.523294 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.523365 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.523312 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.523365 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.523322 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.524723 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.524700 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc6c0f9-ab3c-45e2-9552-e956521e6754-kube-api-access-m2zvl" pod "ecc6c0f9-ab3c-45e2-9552-e956521e6754" (UID: "ecc6c0f9-ab3c-45e2-9552-e956521e6754"). InnerVolumeSpecName "kube-api-access-m2zvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:22:50.524834 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.524819 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-dshm" pod "ecc6c0f9-ab3c-45e2-9552-e956521e6754" (UID: "ecc6c0f9-ab3c-45e2-9552-e956521e6754"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:22:50.624489 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.624449 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tls-certs\") pod \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\" (UID: \"ecc6c0f9-ab3c-45e2-9552-e956521e6754\") " Nov 03 14:22:50.624771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.624756 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc6c0f9-ab3c-45e2-9552-e956521e6754-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.624822 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.624774 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2zvl\" (UniqueName: \"kubernetes.io/projected/ecc6c0f9-ab3c-45e2-9552-e956521e6754-kube-api-access-m2zvl\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.626724 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.626691 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tls-certs" pod "ecc6c0f9-ab3c-45e2-9552-e956521e6754" (UID: "ecc6c0f9-ab3c-45e2-9552-e956521e6754"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:22:50.725765 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:50.725729 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc6c0f9-ab3c-45e2-9552-e956521e6754-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:51.136731 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.136699 4924 generic.go:356] "Generic (PLEG): container finished" podID="ecc6c0f9-ab3c-45e2-9552-e956521e6754" containerID="ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e" exitCode=0 Nov 03 14:22:51.136926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.136748 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" Nov 03 14:22:51.136926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.136787 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" event={"ID":"ecc6c0f9-ab3c-45e2-9552-e956521e6754","Type":"ContainerDied","Data":"ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e"} Nov 03 14:22:51.136926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.136840 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw" event={"ID":"ecc6c0f9-ab3c-45e2-9552-e956521e6754","Type":"ContainerDied","Data":"34dd0ff8a4c6e0f15ff2035bb016d1907937370b9d29ef42c8a039e2e01ce106"} Nov 03 14:22:51.136926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.136863 4924 scope.go:122] "RemoveContainer" containerID="ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e" Nov 03 14:22:51.150439 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.150416 4924 scope.go:122] "RemoveContainer" containerID="ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e" Nov 03 14:22:51.150850 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:22:51.150822 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e\": container with ID starting with ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e not found: ID does not exist" containerID="ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e" Nov 03 14:22:51.150927 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.150865 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e"} err="failed to get container status \"ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e\": rpc error: code = NotFound desc = could not find container \"ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e\": container with ID starting with ff833c2a3847d37a4bf9cc54bdceb9fd106d28a80e0d6443424643fc2a8c9e2e not found: ID does not exist" Nov 03 14:22:51.290560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.290525 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw"] Nov 03 14:22:51.336772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:51.336739 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/rate-test-model-367381-kserve-55946c587c-pqqlw"] Nov 03 14:22:52.561803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:52.561763 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:22:52.951515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:52.951480 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc6c0f9-ab3c-45e2-9552-e956521e6754" path="/var/lib/kubelet/pods/ecc6c0f9-ab3c-45e2-9552-e956521e6754/volumes" Nov 03 14:22:57.660827 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:22:57.660759 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:02.758083 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:02.758041 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:07.938658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:07.938505 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:12.024553 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:12.024496 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:12.915159 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:12.915066 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:18.048898 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:18.048845 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:18.558814 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:23:18.558777 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:23:22.471054 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:22.471015 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:24.263085 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:24.263037 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:29.325146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:29.325106 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:30.719653 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:30.719603 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:30.890933 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:30.890902 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:32.076897 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:32.076855 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:32.296389 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:32.296337 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:32.931386 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:32.931347 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:32.939086 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:32.939041 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl"] Nov 03 14:23:32.939309 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:32.939284 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" containerID="cri-o://28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79" gracePeriod=30 Nov 03 14:23:33.700089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:33.700046 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:34.867169 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:34.867132 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:34.964917 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:34.964874 4924 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:23:34.965102 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:34.964949 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:23:38.215316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.215257 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 14:23:38.342704 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.342668 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-kube-api-access-wsts9\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-kube-api-access-wsts9\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.342903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.342779 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-credential-socket\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.342903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.342823 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32447f3e-2aef-482f-8c7c-08626f6698e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32447f3e-2aef-482f-8c7c-08626f6698e6-istiod-ca-cert\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.342903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.342865 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-certs\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.343066 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.342918 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-token\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-token\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.343066 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.342961 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-socket\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.343066 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.343025 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-podinfo\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.343957 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.343116 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-envoy\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.344327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.343989 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-data\") pod \"32447f3e-2aef-482f-8c7c-08626f6698e6\" (UID: \"32447f3e-2aef-482f-8c7c-08626f6698e6\") " Nov 03 14:23:38.344327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.343307 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-socket" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:38.344327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.343564 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-certs" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:38.344327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.343582 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-credential-socket" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:38.344327 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.343911 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32447f3e-2aef-482f-8c7c-08626f6698e6-istiod-ca-cert" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:23:38.345189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.344621 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.345189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.344759 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32447f3e-2aef-482f-8c7c-08626f6698e6-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.345189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.344778 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.345189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.344781 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-data" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:38.345189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.344793 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.345555 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.345454 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-kube-api-access-wsts9" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "kube-api-access-wsts9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:38.345751 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.345723 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-podinfo" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:23:38.345837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.345813 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-token" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:38.346518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.346498 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-envoy" pod "32447f3e-2aef-482f-8c7c-08626f6698e6" (UID: "32447f3e-2aef-482f-8c7c-08626f6698e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:23:38.446520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.446469 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.446520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.446511 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.446520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.446521 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.446520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.446531 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsts9\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-kube-api-access-wsts9\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.446899 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.446542 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32447f3e-2aef-482f-8c7c-08626f6698e6-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:38.537140 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.537022 4924 generic.go:356] "Generic (PLEG): container finished" podID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerID="28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79" exitCode=0 Nov 03 14:23:38.537140 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.537081 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" Nov 03 14:23:38.537140 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.537108 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" event={"ID":"32447f3e-2aef-482f-8c7c-08626f6698e6","Type":"ContainerDied","Data":"28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79"} Nov 03 14:23:38.537401 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.537208 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl" event={"ID":"32447f3e-2aef-482f-8c7c-08626f6698e6","Type":"ContainerDied","Data":"4dc071e70dcade7163d8afed0bc1fa06806182f5e6fbdc9d8c931ef0878e6a84"} Nov 03 14:23:38.537401 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.537241 4924 scope.go:122] "RemoveContainer" containerID="28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79" Nov 03 14:23:38.551867 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.551844 4924 scope.go:122] "RemoveContainer" containerID="820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c" Nov 03 14:23:38.564346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.564320 4924 scope.go:122] "RemoveContainer" containerID="28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79" Nov 03 14:23:38.564670 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:23:38.564650 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79\": container with ID starting with 28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79 not found: ID does not exist" containerID="28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79" Nov 03 14:23:38.564765 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.564681 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79"} err="failed to get container status \"28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79\": rpc error: code = NotFound desc = could not find container \"28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79\": container with ID starting with 28fcff5a6703c379f5eb316aed02d79b017303735feb2583e67b3626524fab79 not found: ID does not exist" Nov 03 14:23:38.564765 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.564701 4924 scope.go:122] "RemoveContainer" containerID="820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c" Nov 03 14:23:38.564990 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:23:38.564962 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c\": container with ID starting with 820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c not found: ID does not exist" containerID="820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c" Nov 03 14:23:38.565521 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.564997 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c"} err="failed to get container status \"820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c\": rpc error: code = NotFound desc = could not find container \"820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c\": container with ID starting with 820ac4b31507ed02ae4fb04f96e66652b263706e4216507141e450463144686c not found: ID does not exist" Nov 03 14:23:38.692985 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.692951 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl"] Nov 03 14:23:38.699201 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.699167 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5faf5d-openshift-default-699c695fdf-7rptl"] Nov 03 14:23:38.952669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:38.952636 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" path="/var/lib/kubelet/pods/32447f3e-2aef-482f-8c7c-08626f6698e6/volumes" Nov 03 14:23:39.558415 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:39.558377 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:39.687251 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:39.687216 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:40.289211 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:40.289176 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:43.160548 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:43.160513 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:43.291864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:43.291829 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:45.241195 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:45.241162 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:45.701850 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:45.701813 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:48.558030 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:23:48.557984 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:23:50.154714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:50.154678 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:50.276620 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:50.276584 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:50.809370 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:50.809328 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:52.902594 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:52.902513 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7"] Nov 03 14:23:52.920413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:52.920383 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:23:55.908445 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:23:55.908393 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:01.051606 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:01.051569 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:04.046381 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:04.046341 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp"] Nov 03 14:24:04.049282 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:04.049250 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd13b3e3-25gsp"] Nov 03 14:24:04.951001 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:04.950967 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc" path="/var/lib/kubelet/pods/5d0b8e4d-5ce0-48a3-82cd-b8db20efd8fc/volumes" Nov 03 14:24:06.153532 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:06.153487 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:07.925063 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:07.925004 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" podUID="27b63d92-a712-4b7d-b87f-e3429c67107b" containerName="main" containerID="cri-o://52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b" gracePeriod=45 Nov 03 14:24:08.195476 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.195448 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:24:08.309790 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.309755 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-tmp-dir\") pod \"27b63d92-a712-4b7d-b87f-e3429c67107b\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " Nov 03 14:24:08.309790 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.309792 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-home\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-home\") pod \"27b63d92-a712-4b7d-b87f-e3429c67107b\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " Nov 03 14:24:08.310015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.309823 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b63d92-a712-4b7d-b87f-e3429c67107b-kube-api-access-2wgdf\" (UniqueName: \"kubernetes.io/projected/27b63d92-a712-4b7d-b87f-e3429c67107b-kube-api-access-2wgdf\") pod \"27b63d92-a712-4b7d-b87f-e3429c67107b\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " Nov 03 14:24:08.310015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.309860 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-model-cache\") pod \"27b63d92-a712-4b7d-b87f-e3429c67107b\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " Nov 03 14:24:08.310015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.309976 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27b63d92-a712-4b7d-b87f-e3429c67107b-tls-certs\" (UniqueName: \"kubernetes.io/secret/27b63d92-a712-4b7d-b87f-e3429c67107b-tls-certs\") pod \"27b63d92-a712-4b7d-b87f-e3429c67107b\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " Nov 03 14:24:08.310245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.310023 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-dshm\") pod \"27b63d92-a712-4b7d-b87f-e3429c67107b\" (UID: \"27b63d92-a712-4b7d-b87f-e3429c67107b\") " Nov 03 14:24:08.310245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.310096 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-tmp-dir" pod "27b63d92-a712-4b7d-b87f-e3429c67107b" (UID: "27b63d92-a712-4b7d-b87f-e3429c67107b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:08.310245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.310142 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-model-cache" pod "27b63d92-a712-4b7d-b87f-e3429c67107b" (UID: "27b63d92-a712-4b7d-b87f-e3429c67107b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:08.310603 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.310582 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:08.310699 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.310608 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:08.311899 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.311858 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-home" pod "27b63d92-a712-4b7d-b87f-e3429c67107b" (UID: "27b63d92-a712-4b7d-b87f-e3429c67107b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:08.312377 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.312341 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-dshm" pod "27b63d92-a712-4b7d-b87f-e3429c67107b" (UID: "27b63d92-a712-4b7d-b87f-e3429c67107b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:08.312807 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.312780 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b63d92-a712-4b7d-b87f-e3429c67107b-kube-api-access-2wgdf" pod "27b63d92-a712-4b7d-b87f-e3429c67107b" (UID: "27b63d92-a712-4b7d-b87f-e3429c67107b"). InnerVolumeSpecName "kube-api-access-2wgdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:08.316199 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.316172 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b63d92-a712-4b7d-b87f-e3429c67107b-tls-certs" pod "27b63d92-a712-4b7d-b87f-e3429c67107b" (UID: "27b63d92-a712-4b7d-b87f-e3429c67107b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:08.411450 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.411415 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:08.411450 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.411444 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wgdf\" (UniqueName: \"kubernetes.io/projected/27b63d92-a712-4b7d-b87f-e3429c67107b-kube-api-access-2wgdf\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:08.411664 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.411458 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27b63d92-a712-4b7d-b87f-e3429c67107b-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:08.411664 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.411472 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27b63d92-a712-4b7d-b87f-e3429c67107b-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:08.743243 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.743209 4924 generic.go:356] "Generic (PLEG): container finished" podID="27b63d92-a712-4b7d-b87f-e3429c67107b" containerID="52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b" exitCode=0 Nov 03 14:24:08.743458 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.743258 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" Nov 03 14:24:08.743458 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.743297 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" event={"ID":"27b63d92-a712-4b7d-b87f-e3429c67107b","Type":"ContainerDied","Data":"52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b"} Nov 03 14:24:08.743458 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.743346 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7" event={"ID":"27b63d92-a712-4b7d-b87f-e3429c67107b","Type":"ContainerDied","Data":"166b5237c33ad2124d05c50694c847c4f9227f7230b3dd83d4a67fe65bae4ed3"} Nov 03 14:24:08.743458 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.743363 4924 scope.go:122] "RemoveContainer" containerID="52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b" Nov 03 14:24:08.753583 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.753567 4924 scope.go:122] "RemoveContainer" containerID="52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b" Nov 03 14:24:08.753841 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:24:08.753821 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b\": container with ID starting with 52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b not found: ID does not exist" containerID="52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b" Nov 03 14:24:08.753898 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.753851 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b"} err="failed to get container status \"52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b\": rpc error: code = NotFound desc = could not find container \"52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b\": container with ID starting with 52b116fd3cea80f1909a1e5612247734e5c1c867d4b7a5549df5b3fcc284049b not found: ID does not exist" Nov 03 14:24:08.923862 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.923829 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7"] Nov 03 14:24:08.969602 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:08.969569 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2399b8/test-model-2399b8-kserve-847b54869c-bpqx7"] Nov 03 14:24:09.220379 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:09.220346 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:10.951673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:10.951635 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b63d92-a712-4b7d-b87f-e3429c67107b" path="/var/lib/kubelet/pods/27b63d92-a712-4b7d-b87f-e3429c67107b/volumes" Nov 03 14:24:11.270254 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:11.270160 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:16.289926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:16.289891 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:18.560457 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:24:18.560416 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:24:18.629409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:18.629361 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:19.558465 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:19.558420 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:20.014403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:20.014338 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:21.345851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:21.345801 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:25.836307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:25.836268 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:25.965616 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:25.965581 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:26.314522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:26.314467 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:26.560394 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:26.560354 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:27.761290 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:27.761256 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:24:27.782748 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:27.782721 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:24:31.853163 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:31.853115 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:32.149265 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:32.149162 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:36.874966 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:36.874879 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:41.238558 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.238519 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p"] Nov 03 14:24:41.239060 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.238739 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" podUID="eeb824c1-f17f-42bb-b9e8-d0a783069259" containerName="maas-api" containerID="cri-o://4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232" gracePeriod=30 Nov 03 14:24:41.326142 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.326108 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:41.571314 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.571281 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:24:41.721228 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.721196 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls\") pod \"eeb824c1-f17f-42bb-b9e8-d0a783069259\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " Nov 03 14:24:41.721410 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.721257 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-metrics-certs\") pod \"eeb824c1-f17f-42bb-b9e8-d0a783069259\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " Nov 03 14:24:41.721410 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.721348 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb824c1-f17f-42bb-b9e8-d0a783069259-kube-api-access-9xlnr\" (UniqueName: \"kubernetes.io/projected/eeb824c1-f17f-42bb-b9e8-d0a783069259-kube-api-access-9xlnr\") pod \"eeb824c1-f17f-42bb-b9e8-d0a783069259\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " Nov 03 14:24:41.721539 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.721464 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb824c1-f17f-42bb-b9e8-d0a783069259-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb824c1-f17f-42bb-b9e8-d0a783069259-openshift-service-ca\") pod \"eeb824c1-f17f-42bb-b9e8-d0a783069259\" (UID: \"eeb824c1-f17f-42bb-b9e8-d0a783069259\") " Nov 03 14:24:41.721902 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.721870 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb824c1-f17f-42bb-b9e8-d0a783069259-openshift-service-ca" pod "eeb824c1-f17f-42bb-b9e8-d0a783069259" (UID: "eeb824c1-f17f-42bb-b9e8-d0a783069259"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:41.723570 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.723538 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb824c1-f17f-42bb-b9e8-d0a783069259-kube-api-access-9xlnr" pod "eeb824c1-f17f-42bb-b9e8-d0a783069259" (UID: "eeb824c1-f17f-42bb-b9e8-d0a783069259"). InnerVolumeSpecName "kube-api-access-9xlnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:41.723696 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.723643 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls" pod "eeb824c1-f17f-42bb-b9e8-d0a783069259" (UID: "eeb824c1-f17f-42bb-b9e8-d0a783069259"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:41.725060 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.725037 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-metrics-certs" pod "eeb824c1-f17f-42bb-b9e8-d0a783069259" (UID: "eeb824c1-f17f-42bb-b9e8-d0a783069259"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:41.831795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.829901 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlnr\" (UniqueName: \"kubernetes.io/projected/eeb824c1-f17f-42bb-b9e8-d0a783069259-kube-api-access-9xlnr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:41.831795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.829970 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb824c1-f17f-42bb-b9e8-d0a783069259-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:41.831795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.829987 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:41.831795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:41.830001 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb824c1-f17f-42bb-b9e8-d0a783069259-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:42.007801 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.007766 4924 generic.go:356] "Generic (PLEG): container finished" podID="eeb824c1-f17f-42bb-b9e8-d0a783069259" containerID="4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232" exitCode=0 Nov 03 14:24:42.007990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.007822 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" event={"ID":"eeb824c1-f17f-42bb-b9e8-d0a783069259","Type":"ContainerDied","Data":"4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232"} Nov 03 14:24:42.007990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.007826 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" Nov 03 14:24:42.007990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.007865 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p" event={"ID":"eeb824c1-f17f-42bb-b9e8-d0a783069259","Type":"ContainerDied","Data":"0830ad76386045abff68ecb161738f5246e552fcca11123ff84177e080e98533"} Nov 03 14:24:42.007990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.007893 4924 scope.go:122] "RemoveContainer" containerID="4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232" Nov 03 14:24:42.019535 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.019510 4924 scope.go:122] "RemoveContainer" containerID="4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232" Nov 03 14:24:42.019814 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:24:42.019793 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232\": container with ID starting with 4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232 not found: ID does not exist" containerID="4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232" Nov 03 14:24:42.019879 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.019828 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232"} err="failed to get container status \"4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232\": rpc error: code = NotFound desc = could not find container \"4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232\": container with ID starting with 4f31381a4b4f5f9359ceff83d8232706f1ef9a920cfe9ded2a4b1e565b9ad232 not found: ID does not exist" Nov 03 14:24:42.164442 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.164402 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p"] Nov 03 14:24:42.214220 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.214176 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2399b8-6d6f7c74-67q8p"] Nov 03 14:24:42.517423 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.517349 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:42.951426 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:42.951389 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb824c1-f17f-42bb-b9e8-d0a783069259" path="/var/lib/kubelet/pods/eeb824c1-f17f-42bb-b9e8-d0a783069259/volumes" Nov 03 14:24:43.482816 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:43.482781 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:44.319138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:44.319084 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:47.524176 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:47.524136 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:47.649715 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:47.649681 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:48.012556 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.012513 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72"] Nov 03 14:24:48.013052 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013035 4924 cpu_manager.go:414] "containerMap: removing container" podUID="42ead058-3e92-4458-9529-596b2c49e135" containerName="main" Nov 03 14:24:48.013141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013053 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ead058-3e92-4458-9529-596b2c49e135" containerName="main" Nov 03 14:24:48.013141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013095 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ecc6c0f9-ab3c-45e2-9552-e956521e6754" containerName="main" Nov 03 14:24:48.013141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013101 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc6c0f9-ab3c-45e2-9552-e956521e6754" containerName="main" Nov 03 14:24:48.013141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013117 4924 cpu_manager.go:414] "containerMap: removing container" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" Nov 03 14:24:48.013141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013123 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" Nov 03 14:24:48.013141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013141 4924 cpu_manager.go:414] "containerMap: removing container" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" Nov 03 14:24:48.013141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013146 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013157 4924 cpu_manager.go:414] "containerMap: removing container" podUID="27b63d92-a712-4b7d-b87f-e3429c67107b" containerName="main" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013161 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b63d92-a712-4b7d-b87f-e3429c67107b" containerName="main" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013170 4924 cpu_manager.go:414] "containerMap: removing container" podUID="eeb824c1-f17f-42bb-b9e8-d0a783069259" containerName="maas-api" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013174 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb824c1-f17f-42bb-b9e8-d0a783069259" containerName="maas-api" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013244 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b63d92-a712-4b7d-b87f-e3429c67107b" containerName="main" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013252 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb824c1-f17f-42bb-b9e8-d0a783069259" containerName="maas-api" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013259 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ead058-3e92-4458-9529-596b2c49e135" containerName="main" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013267 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc6c0f9-ab3c-45e2-9552-e956521e6754" containerName="main" Nov 03 14:24:48.013498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.013275 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" Nov 03 14:24:48.017688 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.017670 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.092541 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.092500 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:48.114122 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114048 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-dshm\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.114122 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114098 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tmp-dir\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.114433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114231 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgr8\" (UniqueName: \"kubernetes.io/projected/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-kube-api-access-7jgr8\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.114433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114252 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-model-cache\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.114433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114272 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-home\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.114433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114309 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"default-dockercfg-4g28d\"" Nov 03 14:24:48.114433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114344 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"openshift-service-ca.crt\"" Nov 03 14:24:48.114433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114344 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"kube-root-ca.crt\"" Nov 03 14:24:48.114433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114384 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tls-certs\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.114713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.114621 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-367381\"/\"sub-test-model-367381-kserve-self-signed-certs\"" Nov 03 14:24:48.117552 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.117531 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72"] Nov 03 14:24:48.215303 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.215271 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-dshm\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.215303 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.215307 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tmp-dir\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.215559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.215405 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgr8\" (UniqueName: \"kubernetes.io/projected/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-kube-api-access-7jgr8\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.215559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.215434 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-model-cache\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.215559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.215458 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-home\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.215559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.215520 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tls-certs\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.215837 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.215812 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tmp-dir\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.215956 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.215932 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-model-cache\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.216042 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.216020 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-home\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.217755 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.217732 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-dshm\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.218440 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.218421 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tls-certs\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.270771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.270694 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgr8\" (UniqueName: \"kubernetes.io/projected/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-kube-api-access-7jgr8\") pod \"sub-test-model-367381-kserve-759dd59897-97w72\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.330756 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.330711 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:48.573719 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:24:48.562272 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:24:48.715718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:48.715691 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72"] Nov 03 14:24:48.716579 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:24:48.716539 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ddb98a_ef34_45ba_97c4_5fbd2cf6ae6d.slice/crio-461dd36b6a54380be4b67c1f646c1cc86bc9979103f8cf6a417e6886f572ec5a WatchSource:0}: Error finding container 461dd36b6a54380be4b67c1f646c1cc86bc9979103f8cf6a417e6886f572ec5a: Status 404 returned error can't find the container with id 461dd36b6a54380be4b67c1f646c1cc86bc9979103f8cf6a417e6886f572ec5a Nov 03 14:24:49.089125 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:49.085098 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" event={"ID":"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d","Type":"ContainerStarted","Data":"c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963"} Nov 03 14:24:49.089125 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:49.085153 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" event={"ID":"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d","Type":"ContainerStarted","Data":"461dd36b6a54380be4b67c1f646c1cc86bc9979103f8cf6a417e6886f572ec5a"} Nov 03 14:24:49.201734 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:49.201675 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" podStartSLOduration=2.201660257 podStartE2EDuration="2.201660257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:49.197923718 +0000 UTC m=+4822.392327312" watchObservedRunningTime="2026-11-03 14:24:49.201660257 +0000 UTC m=+4822.396063868" Nov 03 14:24:49.248961 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:49.248901 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:49.331980 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:49.331734 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:49.543286 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:49.543239 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:49.886781 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:49.886703 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:50.607030 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:50.606996 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:53.095009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:53.094969 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:53.225530 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:53.225500 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:54.807273 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:54.807237 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:55.118906 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:55.118832 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:55.288039 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:55.288005 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g"] Nov 03 14:24:55.288386 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:55.288353 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" containerID="cri-o://da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc" gracePeriod=30 Nov 03 14:24:55.318535 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:55.318488 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:55.769699 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:55.769662 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:56.879449 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:56.879412 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:57.535640 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:57.535607 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:57.655033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:57.655004 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:24:58.330999 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:58.330963 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:58.345676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:58.345646 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:24:59.164431 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:24:59.164397 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:25:00.578975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.578947 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 14:25:00.675671 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.675636 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-certs\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.675858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.675679 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-data\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.675858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.675780 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-envoy\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.675858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.675820 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-credential-socket\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.676105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.675903 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-socket\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.676105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.675933 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-certs" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:00.676105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.675958 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-token\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-token\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.676105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.675960 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-data" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:00.676105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676007 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istiod-ca-cert\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.676105 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676055 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-kube-api-access-v9dzr\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-kube-api-access-v9dzr\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.676430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676130 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-podinfo\") pod \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\" (UID: \"e94d34a5-63e2-446b-ba23-3a1e80112dc7\") " Nov 03 14:25:00.676430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676225 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-socket" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:00.676430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676240 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-credential-socket" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:00.676607 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676558 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istiod-ca-cert" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:00.676607 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676593 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.676738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676612 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.676738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676621 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.676738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.676634 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.678509 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.678486 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-envoy" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:00.678754 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.678685 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-token" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:00.678754 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.678686 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-podinfo" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:25:00.678754 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.678730 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-kube-api-access-v9dzr" pod "e94d34a5-63e2-446b-ba23-3a1e80112dc7" (UID: "e94d34a5-63e2-446b-ba23-3a1e80112dc7"). InnerVolumeSpecName "kube-api-access-v9dzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:00.740708 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.740675 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm"] Nov 03 14:25:00.777313 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.777276 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.777313 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.777313 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.777522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.777328 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9dzr\" (UniqueName: \"kubernetes.io/projected/e94d34a5-63e2-446b-ba23-3a1e80112dc7-kube-api-access-v9dzr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.777522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.777337 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.777522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.777348 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d34a5-63e2-446b-ba23-3a1e80112dc7-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.791767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.791735 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j"] Nov 03 14:25:00.882405 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:00.882292 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:01.045983 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.045948 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:01.169216 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.169181 4924 generic.go:356] "Generic (PLEG): container finished" podID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerID="da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc" exitCode=0 Nov 03 14:25:01.169429 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.169260 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" Nov 03 14:25:01.169429 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.169263 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" event={"ID":"e94d34a5-63e2-446b-ba23-3a1e80112dc7","Type":"ContainerDied","Data":"da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc"} Nov 03 14:25:01.169429 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.169405 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g" event={"ID":"e94d34a5-63e2-446b-ba23-3a1e80112dc7","Type":"ContainerDied","Data":"f8c0687d44efa90bd8a18c79ae2a9953f5fe00db659e68d0eb3707229cdebb93"} Nov 03 14:25:01.169623 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.169455 4924 scope.go:122] "RemoveContainer" containerID="da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc" Nov 03 14:25:01.180396 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.180370 4924 scope.go:122] "RemoveContainer" containerID="018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b" Nov 03 14:25:01.189466 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.189445 4924 scope.go:122] "RemoveContainer" containerID="da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc" Nov 03 14:25:01.189775 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:25:01.189752 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc\": container with ID starting with da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc not found: ID does not exist" containerID="da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc" Nov 03 14:25:01.189882 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.189788 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc"} err="failed to get container status \"da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc\": rpc error: code = NotFound desc = could not find container \"da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc\": container with ID starting with da6860a063bd300f521532502876349d10378bafa29fc85d0aa04064224dedcc not found: ID does not exist" Nov 03 14:25:01.189882 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.189812 4924 scope.go:122] "RemoveContainer" containerID="018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b" Nov 03 14:25:01.190065 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:25:01.190045 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b\": container with ID starting with 018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b not found: ID does not exist" containerID="018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b" Nov 03 14:25:01.190134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.190096 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b"} err="failed to get container status \"018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b\": rpc error: code = NotFound desc = could not find container \"018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b\": container with ID starting with 018f983ac13a1889c5d1f0330618c123b33774af64558dde1412b982ed82b61b not found: ID does not exist" Nov 03 14:25:01.329332 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.329295 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g"] Nov 03 14:25:01.334794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.334762 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d0231d-openshift-default-dbdc596d9-55d7g"] Nov 03 14:25:01.466433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:01.466344 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:02.779905 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:02.779866 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:02.951892 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:02.951855 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" path="/var/lib/kubelet/pods/e94d34a5-63e2-446b-ba23-3a1e80112dc7/volumes" Nov 03 14:25:06.062861 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:06.062815 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:07.061369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:07.061314 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:07.989795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:07.989755 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:08.660817 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:08.660784 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:11.264414 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:11.264353 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:13.080761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:13.080720 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:14.401470 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.401436 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5"] Nov 03 14:25:14.401943 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.401929 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" Nov 03 14:25:14.401994 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.401943 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" Nov 03 14:25:14.402055 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.402045 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="32447f3e-2aef-482f-8c7c-08626f6698e6" containerName="istio-proxy" Nov 03 14:25:14.402146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.402063 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" Nov 03 14:25:14.402146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.402086 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" Nov 03 14:25:14.405550 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.405528 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.472128 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.472094 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"openshift-service-ca.crt\"" Nov 03 14:25:14.477601 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.477567 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5"] Nov 03 14:25:14.478429 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.478397 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:14.502463 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.502436 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"default-dockercfg-rxrxl\"" Nov 03 14:25:14.502463 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.502464 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"kube-root-ca.crt\"" Nov 03 14:25:14.502463 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.502476 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"sub-test-model-2bceb1-kserve-self-signed-certs\"" Nov 03 14:25:14.526236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.526205 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-tmp-dir\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.526380 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.526275 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-dshm\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.526434 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.526390 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47013a3-ed77-44a3-917b-b05d224cd6f5-tls-certs\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.526479 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.526437 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-home\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.526530 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.526513 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6h8z\" (UniqueName: \"kubernetes.io/projected/d47013a3-ed77-44a3-917b-b05d224cd6f5-kube-api-access-n6h8z\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.526586 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.526576 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-model-cache\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.627520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.627475 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-tmp-dir\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.627709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.627572 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-dshm\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.627709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.627656 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47013a3-ed77-44a3-917b-b05d224cd6f5-tls-certs\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.627843 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.627711 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-home\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.627843 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.627743 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6h8z\" (UniqueName: \"kubernetes.io/projected/d47013a3-ed77-44a3-917b-b05d224cd6f5-kube-api-access-n6h8z\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.627843 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.627817 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-model-cache\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.628049 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.627852 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-tmp-dir\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.628152 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.628108 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-home\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.628214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.628164 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-model-cache\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.629924 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.629897 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-dshm\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.630223 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.630208 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47013a3-ed77-44a3-917b-b05d224cd6f5-tls-certs\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.682044 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.682006 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6h8z\" (UniqueName: \"kubernetes.io/projected/d47013a3-ed77-44a3-917b-b05d224cd6f5-kube-api-access-n6h8z\") pod \"sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.718346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.718312 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:14.910421 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:14.909589 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:15.069096 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:15.068977 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:15.100755 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:15.100719 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5"] Nov 03 14:25:15.109089 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:25:15.109039 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd47013a3_ed77_44a3_917b_b05d224cd6f5.slice/crio-b9a8d08f346020fe789699c63712569e3998ce0036c78e5223fc65f1c1c6d9a4 WatchSource:0}: Error finding container b9a8d08f346020fe789699c63712569e3998ce0036c78e5223fc65f1c1c6d9a4: Status 404 returned error can't find the container with id b9a8d08f346020fe789699c63712569e3998ce0036c78e5223fc65f1c1c6d9a4 Nov 03 14:25:15.265668 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:15.265630 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" event={"ID":"d47013a3-ed77-44a3-917b-b05d224cd6f5","Type":"ContainerStarted","Data":"7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5"} Nov 03 14:25:15.265844 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:15.265678 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" event={"ID":"d47013a3-ed77-44a3-917b-b05d224cd6f5","Type":"ContainerStarted","Data":"b9a8d08f346020fe789699c63712569e3998ce0036c78e5223fc65f1c1c6d9a4"} Nov 03 14:25:15.374388 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:15.374292 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" podStartSLOduration=1.374279942 podStartE2EDuration="1.374279942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:15.372819509 +0000 UTC m=+4848.567223105" watchObservedRunningTime="2026-11-03 14:25:15.374279942 +0000 UTC m=+4848.568683536" Nov 03 14:25:15.719245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:15.719204 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:15.761612 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:15.761567 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerName="main" containerID="cri-o://283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa" gracePeriod=45 Nov 03 14:25:15.812269 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:15.812214 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" containerID="cri-o://82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531" gracePeriod=45 Nov 03 14:25:16.188775 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.188747 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 14:25:16.192324 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.192300 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 14:25:16.273061 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.273027 4924 generic.go:356] "Generic (PLEG): container finished" podID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerID="82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531" exitCode=0 Nov 03 14:25:16.273266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.273102 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" Nov 03 14:25:16.273266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.273111 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" event={"ID":"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42","Type":"ContainerDied","Data":"82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531"} Nov 03 14:25:16.273266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.273155 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j" event={"ID":"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42","Type":"ContainerDied","Data":"ecec620ed3af730a583c87a3d994020ae4f2187a278e41d451d66b884b070103"} Nov 03 14:25:16.273266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.273175 4924 scope.go:122] "RemoveContainer" containerID="82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531" Nov 03 14:25:16.274565 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.274544 4924 generic.go:356] "Generic (PLEG): container finished" podID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerID="283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa" exitCode=0 Nov 03 14:25:16.274666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.274588 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" Nov 03 14:25:16.274666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.274625 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" event={"ID":"d193bf67-8de9-4eb6-b6cc-51d51960cd2b","Type":"ContainerDied","Data":"283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa"} Nov 03 14:25:16.274666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.274657 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" event={"ID":"d193bf67-8de9-4eb6-b6cc-51d51960cd2b","Type":"ContainerDied","Data":"c00ca383be6589dbec6898db6d427103e83a6f0aef06fa64cc2aba61e03b7acf"} Nov 03 14:25:16.283532 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.283513 4924 scope.go:122] "RemoveContainer" containerID="82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531" Nov 03 14:25:16.283785 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:25:16.283769 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531\": container with ID starting with 82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531 not found: ID does not exist" containerID="82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531" Nov 03 14:25:16.283851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.283794 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531"} err="failed to get container status \"82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531\": rpc error: code = NotFound desc = could not find container \"82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531\": container with ID starting with 82c79c918652fd8cfd36c32f4c7805a39ec2d48502750789ef8ca1bc1d83b531 not found: ID does not exist" Nov 03 14:25:16.283851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.283811 4924 scope.go:122] "RemoveContainer" containerID="283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa" Nov 03 14:25:16.294343 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.294305 4924 scope.go:122] "RemoveContainer" containerID="283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa" Nov 03 14:25:16.294709 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:25:16.294684 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa\": container with ID starting with 283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa not found: ID does not exist" containerID="283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa" Nov 03 14:25:16.294794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.294717 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa"} err="failed to get container status \"283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa\": rpc error: code = NotFound desc = could not find container \"283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa\": container with ID starting with 283f7b811073a12e1e90205bdd42af2408e477ff9eca961f962169991c01a6fa not found: ID does not exist" Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335025 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-kube-api-access-lxjkh\" (UniqueName: \"kubernetes.io/projected/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-kube-api-access-lxjkh\") pod \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335102 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tls-certs\") pod \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335182 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-home\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-home\") pod \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335229 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-model-cache\") pod \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335256 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-kube-api-access-rm2pk\" (UniqueName: \"kubernetes.io/projected/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-kube-api-access-rm2pk\") pod \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335314 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tmp-dir\") pod \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335342 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-home\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-home\") pod \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335402 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-model-cache\") pod \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335425 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-dshm\") pod \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335472 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-dshm\") pod \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335514 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tls-certs\") pod \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\" (UID: \"a0494ee2-8d1e-4df7-b26e-8be68d3d1c42\") " Nov 03 14:25:16.336108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.335542 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tmp-dir\") pod \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\" (UID: \"d193bf67-8de9-4eb6-b6cc-51d51960cd2b\") " Nov 03 14:25:16.340360 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.340330 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-model-cache" pod "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" (UID: "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.349218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.345796 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-dshm" pod "d193bf67-8de9-4eb6-b6cc-51d51960cd2b" (UID: "d193bf67-8de9-4eb6-b6cc-51d51960cd2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.349218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.348411 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-dshm" pod "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" (UID: "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.355222 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.354826 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-kube-api-access-rm2pk" pod "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" (UID: "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42"). InnerVolumeSpecName "kube-api-access-rm2pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:16.355222 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.355044 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-home" pod "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" (UID: "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.355630 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.355315 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-model-cache" pod "d193bf67-8de9-4eb6-b6cc-51d51960cd2b" (UID: "d193bf67-8de9-4eb6-b6cc-51d51960cd2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.355630 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.355485 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tls-certs" pod "d193bf67-8de9-4eb6-b6cc-51d51960cd2b" (UID: "d193bf67-8de9-4eb6-b6cc-51d51960cd2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:16.355733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.355711 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tmp-dir" pod "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" (UID: "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.355851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.355827 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tmp-dir" pod "d193bf67-8de9-4eb6-b6cc-51d51960cd2b" (UID: "d193bf67-8de9-4eb6-b6cc-51d51960cd2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.355943 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.355924 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-home" pod "d193bf67-8de9-4eb6-b6cc-51d51960cd2b" (UID: "d193bf67-8de9-4eb6-b6cc-51d51960cd2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.358232 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.356348 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-kube-api-access-lxjkh" pod "d193bf67-8de9-4eb6-b6cc-51d51960cd2b" (UID: "d193bf67-8de9-4eb6-b6cc-51d51960cd2b"). InnerVolumeSpecName "kube-api-access-lxjkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:16.358232 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.357971 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tls-certs" pod "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" (UID: "a0494ee2-8d1e-4df7-b26e-8be68d3d1c42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:16.437560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437526 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437560 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437575 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437588 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437599 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437610 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxjkh\" (UniqueName: \"kubernetes.io/projected/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-kube-api-access-lxjkh\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437622 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437634 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437647 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437660 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm2pk\" (UniqueName: \"kubernetes.io/projected/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-kube-api-access-rm2pk\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437674 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.437808 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.437685 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d193bf67-8de9-4eb6-b6cc-51d51960cd2b-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.701095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.701026 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm"] Nov 03 14:25:16.705499 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.705469 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm"] Nov 03 14:25:16.785601 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.785571 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:16.809126 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.809061 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j"] Nov 03 14:25:16.811855 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.811825 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/premium-w0-42c707-kserve-cf8fcc897-4kg4j"] Nov 03 14:25:16.951848 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.951817 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" path="/var/lib/kubelet/pods/a0494ee2-8d1e-4df7-b26e-8be68d3d1c42/volumes" Nov 03 14:25:16.952264 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:16.952249 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" path="/var/lib/kubelet/pods/d193bf67-8de9-4eb6-b6cc-51d51960cd2b/volumes" Nov 03 14:25:17.064687 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:17.064650 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-2-w0-42c707-kserve-6f884d9c6c-jkzkm" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:25:18.559186 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:25:18.559147 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:25:19.229573 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:19.229506 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:21.873589 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:21.873554 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:24.719143 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:24.719107 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:24.737716 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:24.737683 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:24.993893 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:24.993811 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:25.351964 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:25.351865 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:25:26.956492 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:26.956448 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:28.613710 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:28.613672 4924 scope.go:122] "RemoveContainer" containerID="acac23ac85117de202ad9e9b06238d7107fce2feedfd8abb3c44be2886c93ea1" Nov 03 14:25:32.098114 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:32.098057 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:37.215648 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:37.215579 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:38.471454 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:38.471399 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:38.612092 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:38.608231 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:39.284970 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:39.284928 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:42.255197 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:42.255159 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:43.817189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:43.817120 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:44.148818 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:44.148723 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:44.349394 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:44.349352 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:45.129952 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:45.129895 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:46.709543 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:46.709503 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:48.516546 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:48.516499 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:48.559692 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:25:48.559653 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:25:53.434393 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:53.434355 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:53.624452 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:53.624418 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:55.098474 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:55.098436 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:55.597410 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:55.597379 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:56.536712 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:56.536668 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:56.673668 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:56.673633 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:25:58.531242 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:25:58.531204 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:00.454864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:00.454830 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:05.642878 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:05.642834 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:07.723932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:07.723848 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:07.908226 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:07.908185 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:09.344379 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:09.344339 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:09.651665 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:09.651584 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:09.803637 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:09.803600 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:09.846664 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:09.846629 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr"] Nov 03 14:26:09.846939 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:09.846913 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" containerID="cri-o://e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c" gracePeriod=30 Nov 03 14:26:09.944756 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:09.944722 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:10.981552 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:10.981511 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:13.085064 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:13.085022 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:15.119499 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.119434 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 14:26:15.220979 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.220943 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-certs\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221203 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.220996 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-podinfo\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221203 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221027 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-envoy\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221203 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221053 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-socket\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221207 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-data\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221237 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-kube-api-access-cjbpq\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-kube-api-access-cjbpq\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221308 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-credential-socket\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221315 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-certs" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.221403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221385 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istiod-ca-cert\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221684 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221426 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-token\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-token\") pod \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\" (UID: \"cabbd2d2-2c43-4d4d-aeba-b687168041f0\") " Nov 03 14:26:15.221684 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221468 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-data" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.221684 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221491 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-socket" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.221684 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221592 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-credential-socket" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.221910 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221753 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istiod-ca-cert" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:15.221910 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221892 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.222029 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221915 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.222029 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221932 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.222029 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221948 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.222029 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.221962 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.223479 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.223448 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-podinfo" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:26:15.223586 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.223519 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-kube-api-access-cjbpq" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "kube-api-access-cjbpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:15.223586 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.223523 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-envoy" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.223586 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.223563 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-token" pod "cabbd2d2-2c43-4d4d-aeba-b687168041f0" (UID: "cabbd2d2-2c43-4d4d-aeba-b687168041f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:15.323388 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.323346 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjbpq\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-kube-api-access-cjbpq\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.323388 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.323381 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.323597 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.323395 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.323597 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.323408 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabbd2d2-2c43-4d4d-aeba-b687168041f0-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.700238 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.700204 4924 generic.go:356] "Generic (PLEG): container finished" podID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerID="e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c" exitCode=0 Nov 03 14:26:15.700501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.700259 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" Nov 03 14:26:15.700501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.700286 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" event={"ID":"cabbd2d2-2c43-4d4d-aeba-b687168041f0","Type":"ContainerDied","Data":"e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c"} Nov 03 14:26:15.700501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.700349 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr" event={"ID":"cabbd2d2-2c43-4d4d-aeba-b687168041f0","Type":"ContainerDied","Data":"b5ddc6452653c7c063809a2c278f0b7b094d88c15fe9282c89214e75bb10f44b"} Nov 03 14:26:15.700501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.700382 4924 scope.go:122] "RemoveContainer" containerID="e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c" Nov 03 14:26:15.711212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.711191 4924 scope.go:122] "RemoveContainer" containerID="089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba" Nov 03 14:26:15.719953 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.719930 4924 scope.go:122] "RemoveContainer" containerID="e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c" Nov 03 14:26:15.720251 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:26:15.720226 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c\": container with ID starting with e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c not found: ID does not exist" containerID="e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c" Nov 03 14:26:15.720337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.720264 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c"} err="failed to get container status \"e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c\": rpc error: code = NotFound desc = could not find container \"e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c\": container with ID starting with e3639f1d03ed059dd01d1c66a2906066a0778f13b119b63e9ac927b5eefab64c not found: ID does not exist" Nov 03 14:26:15.720337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.720301 4924 scope.go:122] "RemoveContainer" containerID="089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba" Nov 03 14:26:15.720617 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:26:15.720596 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba\": container with ID starting with 089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba not found: ID does not exist" containerID="089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba" Nov 03 14:26:15.720658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.720624 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba"} err="failed to get container status \"089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba\": rpc error: code = NotFound desc = could not find container \"089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba\": container with ID starting with 089199b6c38df11499c48cad2399633be75a444ebb3dcdf9cefd8ea15007c7ba not found: ID does not exist" Nov 03 14:26:15.868482 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.868446 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr"] Nov 03 14:26:15.913295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:15.913265 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-z5ltr"] Nov 03 14:26:16.367579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:16.367537 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:16.951236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:16.951203 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" path="/var/lib/kubelet/pods/cabbd2d2-2c43-4d4d-aeba-b687168041f0/volumes" Nov 03 14:26:18.559730 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:26:18.559695 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:26:19.571250 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:19.571211 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:21.687340 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:21.687306 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:26.812269 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:26.812234 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:31.959491 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:31.959449 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:37.247381 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:37.247333 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:42.507336 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.507299 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:42.802442 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.802359 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48"] Nov 03 14:26:42.802630 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.802604 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" containerID="cri-o://69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7" gracePeriod=30 Nov 03 14:26:42.850136 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.850085 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76"] Nov 03 14:26:42.850413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.850359 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" containerID="cri-o://f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960" gracePeriod=30 Nov 03 14:26:42.854564 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.854534 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8"] Nov 03 14:26:42.854787 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.854763 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" containerID="cri-o://1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626" gracePeriod=30 Nov 03 14:26:42.894954 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.894920 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk"] Nov 03 14:26:42.895162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:42.895124 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" containerID="cri-o://2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed" gracePeriod=30 Nov 03 14:26:43.632592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:43.632550 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:26:43.633157 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:43.632615 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:26:44.148596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:44.148555 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-kws76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:26:44.148802 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:44.148634 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:26:44.158908 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:44.158875 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:44.293178 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:44.293145 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:44.830319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:44.830271 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-82cq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:26:44.830725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:44.830356 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:26:44.875047 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:44.875006 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-wtf48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:26:44.875307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:44.875066 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:26:47.554044 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:47.553999 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:48.146938 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.146907 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:26:48.177654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.177625 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:26:48.248009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.247978 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:26:48.262552 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.262526 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:26:48.288563 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288533 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-token\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-token\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.288759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288595 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-socket\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.288759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288619 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be04d1a7-2306-4e9b-90c4-354e2521e60f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be04d1a7-2306-4e9b-90c4-354e2521e60f-istiod-ca-cert\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.288759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288649 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-kube-api-access-7g5w9\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-kube-api-access-7g5w9\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.288759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288667 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-data\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.288759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288694 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-data\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.288759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288729 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-istio-token\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-istio-token\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288765 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-socket\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288834 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46fe371-f392-41df-bde9-847cbccef8a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46fe371-f392-41df-bde9-847cbccef8a6-istiod-ca-cert\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288890 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-credential-socket\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288926 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-podinfo\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288914 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-socket" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.288956 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-certs\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289002 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-data" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289031 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-kube-api-access-psssr\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-kube-api-access-psssr\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289126 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-credential-socket\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289135 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be04d1a7-2306-4e9b-90c4-354e2521e60f-istiod-ca-cert" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289239 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-envoy\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289322 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46fe371-f392-41df-bde9-847cbccef8a6-istiod-ca-cert" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289341 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d46fe371-f392-41df-bde9-847cbccef8a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46fe371-f392-41df-bde9-847cbccef8a6-istio-podinfo\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289360 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-credential-socket" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289373 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-certs\") pod \"d46fe371-f392-41df-bde9-847cbccef8a6\" (UID: \"d46fe371-f392-41df-bde9-847cbccef8a6\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289410 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-envoy\") pod \"be04d1a7-2306-4e9b-90c4-354e2521e60f\" (UID: \"be04d1a7-2306-4e9b-90c4-354e2521e60f\") " Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289530 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-certs" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289607 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-data" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289761 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-credential-socket" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.289931 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-certs" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290260 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290283 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290301 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be04d1a7-2306-4e9b-90c4-354e2521e60f-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290316 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290332 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290348 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d46fe371-f392-41df-bde9-847cbccef8a6-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290362 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.290369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290376 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.292091 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290391 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.292091 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.290456 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-socket" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.292091 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.291518 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-token" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.292091 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.291654 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-istio-token" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.292091 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.291819 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-kube-api-access-7g5w9" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "kube-api-access-7g5w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.292299 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.292224 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-envoy" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.292299 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.292228 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-podinfo" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:26:48.292438 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.292420 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-kube-api-access-psssr" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "kube-api-access-psssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.292935 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.292917 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-envoy" pod "be04d1a7-2306-4e9b-90c4-354e2521e60f" (UID: "be04d1a7-2306-4e9b-90c4-354e2521e60f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.293407 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.293391 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d46fe371-f392-41df-bde9-847cbccef8a6-istio-podinfo" pod "d46fe371-f392-41df-bde9-847cbccef8a6" (UID: "d46fe371-f392-41df-bde9-847cbccef8a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:26:48.390971 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.390876 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a63cdbb-c946-4c40-955d-05168264ef47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a63cdbb-c946-4c40-955d-05168264ef47-istiod-ca-cert\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.390971 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.390943 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-data\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.390971 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.390961 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-socket\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.391316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391100 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-certs\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.391316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391156 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-token\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-token\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.391316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391208 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-podinfo\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.391316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391248 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-envoy\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.391316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391225 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-socket" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.391316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391289 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-data" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.391316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391304 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-kube-api-access-tl7wn\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-kube-api-access-tl7wn\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391369 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-certs\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391397 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-data\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391396 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-certs" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391440 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9713196-25d4-4c50-9327-2d4c1010baf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9713196-25d4-4c50-9327-2d4c1010baf6-istiod-ca-cert\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391438 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a63cdbb-c946-4c40-955d-05168264ef47-istiod-ca-cert" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391476 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-kube-api-access-blj4b\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-kube-api-access-blj4b\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391526 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-socket\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391570 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-credential-socket\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391614 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-credential-socket\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391642 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-certs" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391645 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-data" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391667 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a63cdbb-c946-4c40-955d-05168264ef47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a63cdbb-c946-4c40-955d-05168264ef47-istio-podinfo\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.391709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391716 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-istio-token\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-istio-token\") pod \"5a63cdbb-c946-4c40-955d-05168264ef47\" (UID: \"5a63cdbb-c946-4c40-955d-05168264ef47\") " Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391747 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-envoy\") pod \"f9713196-25d4-4c50-9327-2d4c1010baf6\" (UID: \"f9713196-25d4-4c50-9327-2d4c1010baf6\") " Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391828 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-socket" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.391962 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9713196-25d4-4c50-9327-2d4c1010baf6-istiod-ca-cert" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392205 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392228 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392244 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psssr\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-kube-api-access-psssr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392259 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392275 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392288 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392303 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9713196-25d4-4c50-9327-2d4c1010baf6-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392319 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d46fe371-f392-41df-bde9-847cbccef8a6-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392332 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392346 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392360 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be04d1a7-2306-4e9b-90c4-354e2521e60f-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.392472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392460 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-credential-socket" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.393315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392485 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g5w9\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-kube-api-access-7g5w9\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.393315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392503 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d46fe371-f392-41df-bde9-847cbccef8a6-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.393315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392517 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d46fe371-f392-41df-bde9-847cbccef8a6-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.393315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392531 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a63cdbb-c946-4c40-955d-05168264ef47-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.393315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392544 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.393315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392558 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.393315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.392647 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-credential-socket" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.394000 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.393964 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-envoy" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.394175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.394065 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-token" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.394175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.394120 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-kube-api-access-tl7wn" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "kube-api-access-tl7wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.394175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.394147 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-kube-api-access-blj4b" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "kube-api-access-blj4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.394530 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.394510 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-podinfo" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:26:48.394658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.394646 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a63cdbb-c946-4c40-955d-05168264ef47-istio-podinfo" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:26:48.395275 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.395256 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-envoy" pod "f9713196-25d4-4c50-9327-2d4c1010baf6" (UID: "f9713196-25d4-4c50-9327-2d4c1010baf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.395338 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.395294 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-istio-token" pod "5a63cdbb-c946-4c40-955d-05168264ef47" (UID: "5a63cdbb-c946-4c40-955d-05168264ef47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.493319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493281 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493313 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493326 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493337 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl7wn\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-kube-api-access-tl7wn\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493350 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj4b\" (UniqueName: \"kubernetes.io/projected/f9713196-25d4-4c50-9327-2d4c1010baf6-kube-api-access-blj4b\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493361 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a63cdbb-c946-4c40-955d-05168264ef47-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493371 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493380 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a63cdbb-c946-4c40-955d-05168264ef47-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493389 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a63cdbb-c946-4c40-955d-05168264ef47-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.493560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.493398 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9713196-25d4-4c50-9327-2d4c1010baf6-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.569947 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:26:48.567679 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:26:48.920517 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.920483 4924 generic.go:356] "Generic (PLEG): container finished" podID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerID="1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626" exitCode=0 Nov 03 14:26:48.920717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.920528 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" Nov 03 14:26:48.920717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.920568 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" event={"ID":"f9713196-25d4-4c50-9327-2d4c1010baf6","Type":"ContainerDied","Data":"1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626"} Nov 03 14:26:48.920717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.920627 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8" event={"ID":"f9713196-25d4-4c50-9327-2d4c1010baf6","Type":"ContainerDied","Data":"cd2981f2e242939a3a1e025276c71e56255846a148a5730105bd40398a3e7ce6"} Nov 03 14:26:48.920717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.920656 4924 scope.go:122] "RemoveContainer" containerID="1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626" Nov 03 14:26:48.922155 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.922124 4924 generic.go:356] "Generic (PLEG): container finished" podID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerID="69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7" exitCode=0 Nov 03 14:26:48.922244 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.922164 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" Nov 03 14:26:48.922244 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.922211 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" event={"ID":"be04d1a7-2306-4e9b-90c4-354e2521e60f","Type":"ContainerDied","Data":"69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7"} Nov 03 14:26:48.922357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.922260 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48" event={"ID":"be04d1a7-2306-4e9b-90c4-354e2521e60f","Type":"ContainerDied","Data":"37412ffeba26c3eaff17567f620554f20413435c47ddc93ccffa868205025241"} Nov 03 14:26:48.932147 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.932120 4924 generic.go:356] "Generic (PLEG): container finished" podID="d46fe371-f392-41df-bde9-847cbccef8a6" containerID="f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960" exitCode=0 Nov 03 14:26:48.932267 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.932151 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" Nov 03 14:26:48.932335 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.932272 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" event={"ID":"d46fe371-f392-41df-bde9-847cbccef8a6","Type":"ContainerDied","Data":"f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960"} Nov 03 14:26:48.932335 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.932313 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76" event={"ID":"d46fe371-f392-41df-bde9-847cbccef8a6","Type":"ContainerDied","Data":"e8cb42f3ac334340f38b674453e7a517780d239983554b106168ace82c0216bd"} Nov 03 14:26:48.934606 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.934584 4924 generic.go:356] "Generic (PLEG): container finished" podID="5a63cdbb-c946-4c40-955d-05168264ef47" containerID="2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed" exitCode=0 Nov 03 14:26:48.934727 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.934644 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" Nov 03 14:26:48.934727 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.934676 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" event={"ID":"5a63cdbb-c946-4c40-955d-05168264ef47","Type":"ContainerDied","Data":"2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed"} Nov 03 14:26:48.934727 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.934710 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk" event={"ID":"5a63cdbb-c946-4c40-955d-05168264ef47","Type":"ContainerDied","Data":"b9351d3818009950cbc4cf682ce28be0e9bed9f728e7d724fb06d2d9fa610270"} Nov 03 14:26:48.938149 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.938129 4924 scope.go:122] "RemoveContainer" containerID="1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626" Nov 03 14:26:48.938465 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:26:48.938441 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626\": container with ID starting with 1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626 not found: ID does not exist" containerID="1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626" Nov 03 14:26:48.938580 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.938476 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626"} err="failed to get container status \"1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626\": rpc error: code = NotFound desc = could not find container \"1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626\": container with ID starting with 1c8783521d55b2fe33e336dcd273687b61a69010c4e0f802a7f3bdfa30cb5626 not found: ID does not exist" Nov 03 14:26:48.938580 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.938505 4924 scope.go:122] "RemoveContainer" containerID="69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7" Nov 03 14:26:48.951305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.951282 4924 scope.go:122] "RemoveContainer" containerID="69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7" Nov 03 14:26:48.951601 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:26:48.951576 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7\": container with ID starting with 69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7 not found: ID does not exist" containerID="69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7" Nov 03 14:26:48.951669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.951610 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7"} err="failed to get container status \"69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7\": rpc error: code = NotFound desc = could not find container \"69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7\": container with ID starting with 69b3a2d2f44686b50d0568fa7bacaaa2cf4002d11db0fe4db04e874b21f610a7 not found: ID does not exist" Nov 03 14:26:48.951669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.951633 4924 scope.go:122] "RemoveContainer" containerID="f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960" Nov 03 14:26:48.961947 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.961797 4924 scope.go:122] "RemoveContainer" containerID="f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960" Nov 03 14:26:48.962101 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:26:48.962065 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960\": container with ID starting with f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960 not found: ID does not exist" containerID="f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960" Nov 03 14:26:48.962148 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.962111 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960"} err="failed to get container status \"f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960\": rpc error: code = NotFound desc = could not find container \"f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960\": container with ID starting with f1be95445a78ad563032b78479d515c0c22f398173f7ca1d622ea2bf0cc25960 not found: ID does not exist" Nov 03 14:26:48.962148 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.962132 4924 scope.go:122] "RemoveContainer" containerID="2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed" Nov 03 14:26:48.971247 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.971226 4924 scope.go:122] "RemoveContainer" containerID="2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed" Nov 03 14:26:48.982166 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:26:48.982133 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed\": container with ID starting with 2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed not found: ID does not exist" containerID="2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed" Nov 03 14:26:48.982291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:48.982174 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed"} err="failed to get container status \"2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed\": rpc error: code = NotFound desc = could not find container \"2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed\": container with ID starting with 2a95f6ad8a7552d47fa6e1afb2e87bcc0e9c7323a88123c55e831644c2b53bed not found: ID does not exist" Nov 03 14:26:49.039539 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.039501 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w"] Nov 03 14:26:49.040124 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040104 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" Nov 03 14:26:49.040124 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040124 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040138 4924 cpu_manager.go:414] "containerMap: removing container" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040143 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040152 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040157 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040166 4924 cpu_manager.go:414] "containerMap: removing container" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040171 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040177 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040183 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040190 4924 cpu_manager.go:414] "containerMap: removing container" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040195 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040213 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerName="main" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040217 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerName="main" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040227 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" Nov 03 14:26:49.040229 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040232 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94d34a5-63e2-446b-ba23-3a1e80112dc7" containerName="istio-proxy" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040243 4924 cpu_manager.go:414] "containerMap: removing container" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040248 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040337 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d193bf67-8de9-4eb6-b6cc-51d51960cd2b" containerName="main" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040347 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" containerName="istio-proxy" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040355 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" containerName="istio-proxy" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040361 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040367 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0494ee2-8d1e-4df7-b26e-8be68d3d1c42" containerName="main" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040372 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" containerName="istio-proxy" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040377 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" containerName="istio-proxy" Nov 03 14:26:49.040661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.040383 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabbd2d2-2c43-4d4d-aeba-b687168041f0" containerName="istio-proxy" Nov 03 14:26:49.045365 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.045347 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.078917 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.078833 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76"] Nov 03 14:26:49.082696 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.082609 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-kws76"] Nov 03 14:26:49.100592 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.100564 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w"] Nov 03 14:26:49.112830 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.112798 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"kube-root-ca.crt\"" Nov 03 14:26:49.112830 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.112820 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"openshift-service-ca.crt\"" Nov 03 14:26:49.121544 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.121523 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"sub-test-model-d341af-kserve-self-signed-certs\"" Nov 03 14:26:49.121670 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.121645 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"default-dockercfg-hln72\"" Nov 03 14:26:49.128599 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.128570 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:49.190159 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.190124 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk"] Nov 03 14:26:49.193502 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.193476 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lgnwk"] Nov 03 14:26:49.200681 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.200649 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-tmp-dir\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.200804 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.200702 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmwg\" (UniqueName: \"kubernetes.io/projected/b26d3697-b507-4e1b-9add-ceca7dac937a-kube-api-access-qtmwg\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.200846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.200823 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d3697-b507-4e1b-9add-ceca7dac937a-tls-certs\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.200846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.200843 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-dshm\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.200928 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.200871 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-home\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.200988 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.200961 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-model-cache\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.300678 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.300635 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8"] Nov 03 14:26:49.302156 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.302121 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-home\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.302304 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.302180 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-model-cache\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.302304 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.302298 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-tmp-dir\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.302462 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.302343 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmwg\" (UniqueName: \"kubernetes.io/projected/b26d3697-b507-4e1b-9add-ceca7dac937a-kube-api-access-qtmwg\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.302462 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.302432 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d3697-b507-4e1b-9add-ceca7dac937a-tls-certs\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.302462 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.302456 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-dshm\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.302641 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.302620 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-home\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.302778 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.302748 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-model-cache\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.303118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.303095 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-tmp-dir\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.305445 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.305413 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-dshm\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.305807 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.305785 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d3697-b507-4e1b-9add-ceca7dac937a-tls-certs\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.305964 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.305948 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-82cq8"] Nov 03 14:26:49.361593 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.361515 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmwg\" (UniqueName: \"kubernetes.io/projected/b26d3697-b507-4e1b-9add-ceca7dac937a-kube-api-access-qtmwg\") pod \"sub-test-model-d341af-kserve-6c4cc7659f-67b2w\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:49.416287 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.416245 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48"] Nov 03 14:26:49.419709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.419679 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-wtf48"] Nov 03 14:26:49.658650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:49.658616 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:50.033301 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:50.033275 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w"] Nov 03 14:26:50.034036 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:26:50.034010 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb26d3697_b507_4e1b_9add_ceca7dac937a.slice/crio-31ea555b9734c35f61a6938a6ddd64be8a3d62b2233c2a6c388211390c6fec06 WatchSource:0}: Error finding container 31ea555b9734c35f61a6938a6ddd64be8a3d62b2233c2a6c388211390c6fec06: Status 404 returned error can't find the container with id 31ea555b9734c35f61a6938a6ddd64be8a3d62b2233c2a6c388211390c6fec06 Nov 03 14:26:50.950266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:50.950234 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a63cdbb-c946-4c40-955d-05168264ef47" path="/var/lib/kubelet/pods/5a63cdbb-c946-4c40-955d-05168264ef47/volumes" Nov 03 14:26:50.950800 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:50.950782 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be04d1a7-2306-4e9b-90c4-354e2521e60f" path="/var/lib/kubelet/pods/be04d1a7-2306-4e9b-90c4-354e2521e60f/volumes" Nov 03 14:26:50.951381 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:50.951364 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46fe371-f392-41df-bde9-847cbccef8a6" path="/var/lib/kubelet/pods/d46fe371-f392-41df-bde9-847cbccef8a6/volumes" Nov 03 14:26:50.952241 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:50.952223 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9713196-25d4-4c50-9327-2d4c1010baf6" path="/var/lib/kubelet/pods/f9713196-25d4-4c50-9327-2d4c1010baf6/volumes" Nov 03 14:26:50.952527 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:50.952509 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" event={"ID":"b26d3697-b507-4e1b-9add-ceca7dac937a","Type":"ContainerStarted","Data":"e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff"} Nov 03 14:26:50.952590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:50.952537 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" event={"ID":"b26d3697-b507-4e1b-9add-ceca7dac937a","Type":"ContainerStarted","Data":"31ea555b9734c35f61a6938a6ddd64be8a3d62b2233c2a6c388211390c6fec06"} Nov 03 14:26:51.072235 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:51.072175 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" podStartSLOduration=3.072157903 podStartE2EDuration="3.072157903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:51.068910395 +0000 UTC m=+4944.263313988" watchObservedRunningTime="2026-11-03 14:26:51.072157903 +0000 UTC m=+4944.266561497" Nov 03 14:26:51.658928 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:51.658893 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:52.859962 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:52.859920 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:58.992877 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:58.992829 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:26:59.659392 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:59.659349 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:26:59.673083 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:26:59.673041 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:27:00.022767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:00.022682 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:27:03.214909 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:03.214864 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:08.535300 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:08.535232 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:14.132267 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:14.132228 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:17.048316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:17.048269 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:17.223550 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:17.223513 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:18.558671 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:27:18.558628 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:27:19.221471 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:19.221435 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:24.473603 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:24.473556 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:29.662933 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:29.662890 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:34.777054 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:34.777021 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:39.976825 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:39.976785 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:40.578742 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:40.578701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:40.736347 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:40.736310 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:43.517749 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:43.517718 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:44.488115 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:44.488079 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:45.222682 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:45.222645 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:48.559226 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:27:48.559186 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:27:49.238685 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:49.238635 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:49.377604 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:49.377558 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:50.557605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:50.557569 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:50.679324 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:50.679288 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:55.879802 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:55.879765 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:27:56.103707 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:27:56.103650 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:00.982480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:00.982433 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:06.091975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:06.091881 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:11.284933 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:11.284897 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:12.348037 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:12.347980 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:16.404650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:16.404612 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:18.558860 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:28:18.558820 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:28:20.771681 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:20.771640 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv"] Nov 03 14:28:20.772144 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:20.771905 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" podUID="311839e4-3e8a-4e9c-be23-98239fc44c41" containerName="maas-api" containerID="cri-o://1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24" gracePeriod=30 Nov 03 14:28:20.858859 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:20.858820 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:21.057979 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.057938 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 14:28:21.146706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.146669 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311839e4-3e8a-4e9c-be23-98239fc44c41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311839e4-3e8a-4e9c-be23-98239fc44c41-openshift-service-ca\") pod \"311839e4-3e8a-4e9c-be23-98239fc44c41\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " Nov 03 14:28:21.146895 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.146731 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls\") pod \"311839e4-3e8a-4e9c-be23-98239fc44c41\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " Nov 03 14:28:21.147008 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.146983 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-metrics-certs\") pod \"311839e4-3e8a-4e9c-be23-98239fc44c41\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " Nov 03 14:28:21.147120 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.147033 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311839e4-3e8a-4e9c-be23-98239fc44c41-kube-api-access-j6sqc\" (UniqueName: \"kubernetes.io/projected/311839e4-3e8a-4e9c-be23-98239fc44c41-kube-api-access-j6sqc\") pod \"311839e4-3e8a-4e9c-be23-98239fc44c41\" (UID: \"311839e4-3e8a-4e9c-be23-98239fc44c41\") " Nov 03 14:28:21.147194 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.147135 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311839e4-3e8a-4e9c-be23-98239fc44c41-openshift-service-ca" pod "311839e4-3e8a-4e9c-be23-98239fc44c41" (UID: "311839e4-3e8a-4e9c-be23-98239fc44c41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:21.147389 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.147367 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311839e4-3e8a-4e9c-be23-98239fc44c41-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:21.149122 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.149062 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-metrics-certs" pod "311839e4-3e8a-4e9c-be23-98239fc44c41" (UID: "311839e4-3e8a-4e9c-be23-98239fc44c41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:21.149453 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.149431 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls" pod "311839e4-3e8a-4e9c-be23-98239fc44c41" (UID: "311839e4-3e8a-4e9c-be23-98239fc44c41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:21.149644 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.149627 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311839e4-3e8a-4e9c-be23-98239fc44c41-kube-api-access-j6sqc" pod "311839e4-3e8a-4e9c-be23-98239fc44c41" (UID: "311839e4-3e8a-4e9c-be23-98239fc44c41"). InnerVolumeSpecName "kube-api-access-j6sqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:21.248019 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.247985 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:21.248019 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.248015 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311839e4-3e8a-4e9c-be23-98239fc44c41-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:21.248019 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.248026 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6sqc\" (UniqueName: \"kubernetes.io/projected/311839e4-3e8a-4e9c-be23-98239fc44c41-kube-api-access-j6sqc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:21.534900 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.534860 4924 generic.go:356] "Generic (PLEG): container finished" podID="311839e4-3e8a-4e9c-be23-98239fc44c41" containerID="1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24" exitCode=0 Nov 03 14:28:21.535240 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.534912 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" Nov 03 14:28:21.535240 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.534943 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" event={"ID":"311839e4-3e8a-4e9c-be23-98239fc44c41","Type":"ContainerDied","Data":"1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24"} Nov 03 14:28:21.535240 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.535001 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv" event={"ID":"311839e4-3e8a-4e9c-be23-98239fc44c41","Type":"ContainerDied","Data":"f42d5a8a0f322e8fe72e22f9eecb1b32a7640a9ed97662ed2dcf590af0187f37"} Nov 03 14:28:21.535240 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.535025 4924 scope.go:122] "RemoveContainer" containerID="1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24" Nov 03 14:28:21.545185 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.545169 4924 scope.go:122] "RemoveContainer" containerID="1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24" Nov 03 14:28:21.545438 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:28:21.545418 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24\": container with ID starting with 1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24 not found: ID does not exist" containerID="1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24" Nov 03 14:28:21.545479 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.545457 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24"} err="failed to get container status \"1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24\": rpc error: code = NotFound desc = could not find container \"1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24\": container with ID starting with 1dc169f7256c06f626a457c3b79a8f45acb474e59040a65103f338b3118cee24 not found: ID does not exist" Nov 03 14:28:21.697584 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.697537 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv"] Nov 03 14:28:21.697793 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:21.697606 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5ae63d-6df479d99f-6mwnv"] Nov 03 14:28:22.677963 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:22.677920 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:22.950029 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:22.949947 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311839e4-3e8a-4e9c-be23-98239fc44c41" path="/var/lib/kubelet/pods/311839e4-3e8a-4e9c-be23-98239fc44c41/volumes" Nov 03 14:28:27.767767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:27.767718 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:28.473343 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:28.473301 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:28.610133 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:28.610092 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:30.116272 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:30.116240 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:30.630718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:30.630682 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:30.861759 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:30.861727 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:31.001510 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:31.001477 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:31.437980 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:31.437922 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:32.367109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:32.367044 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:32.378554 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:32.378517 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72"] Nov 03 14:28:32.820118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:32.820062 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:33.449911 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:33.449853 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:37.226653 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:37.226620 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:37.348688 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:37.348656 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:37.717631 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:37.717594 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:38.902292 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:38.902250 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:39.229473 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:39.229437 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:39.683346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:39.683311 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:39.803407 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:39.803374 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:43.746379 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:43.746309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:44.576715 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:44.576676 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:44.803923 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:44.803857 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:44.923656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:44.923620 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:47.398542 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.398498 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" podUID="55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" containerName="main" containerID="cri-o://c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963" gracePeriod=45 Nov 03 14:28:47.656335 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.656264 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:28:47.716722 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.716685 4924 generic.go:356] "Generic (PLEG): container finished" podID="55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" containerID="c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963" exitCode=0 Nov 03 14:28:47.716926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.716735 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" Nov 03 14:28:47.716926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.716777 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" event={"ID":"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d","Type":"ContainerDied","Data":"c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963"} Nov 03 14:28:47.716926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.716830 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72" event={"ID":"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d","Type":"ContainerDied","Data":"461dd36b6a54380be4b67c1f646c1cc86bc9979103f8cf6a417e6886f572ec5a"} Nov 03 14:28:47.716926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.716853 4924 scope.go:122] "RemoveContainer" containerID="c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963" Nov 03 14:28:47.726218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.726197 4924 scope.go:122] "RemoveContainer" containerID="c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963" Nov 03 14:28:47.726536 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:28:47.726510 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963\": container with ID starting with c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963 not found: ID does not exist" containerID="c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963" Nov 03 14:28:47.726681 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.726544 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963"} err="failed to get container status \"c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963\": rpc error: code = NotFound desc = could not find container \"c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963\": container with ID starting with c68488f6c2bc7052228576ccc02fbef4e108b60ee9c5cb9931f013e40bcd1963 not found: ID does not exist" Nov 03 14:28:47.824775 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.824744 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tls-certs\") pod \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " Nov 03 14:28:47.824984 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.824795 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-model-cache\") pod \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " Nov 03 14:28:47.824984 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.824852 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-kube-api-access-7jgr8\" (UniqueName: \"kubernetes.io/projected/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-kube-api-access-7jgr8\") pod \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " Nov 03 14:28:47.824984 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.824948 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-home\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-home\") pod \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " Nov 03 14:28:47.825196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.824987 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-dshm\") pod \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " Nov 03 14:28:47.825196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.825013 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tmp-dir\") pod \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\" (UID: \"55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d\") " Nov 03 14:28:47.825196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.825104 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-model-cache" pod "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" (UID: "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:47.825326 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.825200 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-home" pod "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" (UID: "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:47.825392 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.825352 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tmp-dir" pod "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" (UID: "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:47.825452 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.825433 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:47.825452 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.825446 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:47.825565 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.825478 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:47.827047 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.827022 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-kube-api-access-7jgr8" pod "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" (UID: "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d"). InnerVolumeSpecName "kube-api-access-7jgr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:47.827202 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.827123 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tls-certs" pod "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" (UID: "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:47.827412 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.827383 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-dshm" pod "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" (UID: "55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:47.926262 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.926225 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:47.926262 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.926256 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:47.926262 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:47.926266 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgr8\" (UniqueName: \"kubernetes.io/projected/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d-kube-api-access-7jgr8\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:48.134782 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:48.134740 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72"] Nov 03 14:28:48.137548 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:48.137516 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-367381/sub-test-model-367381-kserve-759dd59897-97w72"] Nov 03 14:28:48.558106 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:28:48.558056 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:28:48.950641 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:48.950606 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" path="/var/lib/kubelet/pods/55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d/volumes" Nov 03 14:28:49.267815 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:49.267735 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:51.174296 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:51.174257 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:56.267979 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:56.267938 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:59.344956 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:59.344914 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:28:59.474421 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:28:59.474383 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:00.976932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:00.976888 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:01.458903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:01.458853 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:01.680762 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:01.680671 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:02.280415 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:02.280374 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:05.333549 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.333514 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-703b9c"] Nov 03 14:29:05.333961 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.333947 4924 cpu_manager.go:414] "containerMap: removing container" podUID="55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" containerName="main" Nov 03 14:29:05.334007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.333962 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" containerName="main" Nov 03 14:29:05.334007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.333994 4924 cpu_manager.go:414] "containerMap: removing container" podUID="311839e4-3e8a-4e9c-be23-98239fc44c41" containerName="maas-api" Nov 03 14:29:05.334007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.333999 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311839e4-3e8a-4e9c-be23-98239fc44c41" containerName="maas-api" Nov 03 14:29:05.334131 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.334088 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ddb98a-ef34-45ba-97c4-5fbd2cf6ae6d" containerName="main" Nov 03 14:29:05.334131 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.334105 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="311839e4-3e8a-4e9c-be23-98239fc44c41" containerName="maas-api" Nov 03 14:29:05.338908 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.338887 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:05.387780 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.387750 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-703b9c"] Nov 03 14:29:05.394630 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.394602 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85x4g\" (UniqueName: \"kubernetes.io/projected/c4eb0d94-96ef-4851-af45-255650fb3f91-kube-api-access-85x4g\") pod \"test-curl-7600-703b9c\" (UID: \"c4eb0d94-96ef-4851-af45-255650fb3f91\") " pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:05.431196 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.431168 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kmbtk\"" Nov 03 14:29:05.496342 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.496307 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85x4g\" (UniqueName: \"kubernetes.io/projected/c4eb0d94-96ef-4851-af45-255650fb3f91-kube-api-access-85x4g\") pod \"test-curl-7600-703b9c\" (UID: \"c4eb0d94-96ef-4851-af45-255650fb3f91\") " pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:05.549847 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.549814 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85x4g\" (UniqueName: \"kubernetes.io/projected/c4eb0d94-96ef-4851-af45-255650fb3f91-kube-api-access-85x4g\") pod \"test-curl-7600-703b9c\" (UID: \"c4eb0d94-96ef-4851-af45-255650fb3f91\") " pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:05.650852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:05.650765 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:06.020042 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:06.020006 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-703b9c"] Nov 03 14:29:06.021534 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:29:06.021505 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4eb0d94_96ef_4851_af45_255650fb3f91.slice/crio-417adc90edd2123b6e1acf8da856a9aca9bf507ed01bd3322705c8d868ae61bb WatchSource:0}: Error finding container 417adc90edd2123b6e1acf8da856a9aca9bf507ed01bd3322705c8d868ae61bb: Status 404 returned error can't find the container with id 417adc90edd2123b6e1acf8da856a9aca9bf507ed01bd3322705c8d868ae61bb Nov 03 14:29:06.832339 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:06.832303 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-703b9c" event={"ID":"c4eb0d94-96ef-4851-af45-255650fb3f91","Type":"ContainerStarted","Data":"6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce"} Nov 03 14:29:06.832828 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:06.832348 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-703b9c" event={"ID":"c4eb0d94-96ef-4851-af45-255650fb3f91","Type":"ContainerStarted","Data":"417adc90edd2123b6e1acf8da856a9aca9bf507ed01bd3322705c8d868ae61bb"} Nov 03 14:29:06.939833 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:06.939739 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-703b9c" podStartSLOduration=1.939724641 podStartE2EDuration="1.939724641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:06.938666861 +0000 UTC m=+5080.133070482" watchObservedRunningTime="2026-11-03 14:29:06.939724641 +0000 UTC m=+5080.134128234" Nov 03 14:29:07.177195 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:07.177156 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:08.377719 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.377681 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-703b9c"] Nov 03 14:29:08.384735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.384706 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-703b9c"] Nov 03 14:29:08.529658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.529626 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-c9275c"] Nov 03 14:29:08.530273 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.530214 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c4eb0d94-96ef-4851-af45-255650fb3f91" containerName="test-curl-7600-703b9c" Nov 03 14:29:08.530273 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.530266 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4eb0d94-96ef-4851-af45-255650fb3f91" containerName="test-curl-7600-703b9c" Nov 03 14:29:08.530520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.530430 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4eb0d94-96ef-4851-af45-255650fb3f91" containerName="test-curl-7600-703b9c" Nov 03 14:29:08.534604 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.534584 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:08.586598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.586569 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-c9275c"] Nov 03 14:29:08.625609 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.625575 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdkbd\" (UniqueName: \"kubernetes.io/projected/e154ec1f-76b5-4d6a-8548-e2f43f31a754-kube-api-access-kdkbd\") pod \"test-curl-7600-c9275c\" (UID: \"e154ec1f-76b5-4d6a-8548-e2f43f31a754\") " pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:08.726244 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.726203 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdkbd\" (UniqueName: \"kubernetes.io/projected/e154ec1f-76b5-4d6a-8548-e2f43f31a754-kube-api-access-kdkbd\") pod \"test-curl-7600-c9275c\" (UID: \"e154ec1f-76b5-4d6a-8548-e2f43f31a754\") " pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:08.779980 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.779940 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdkbd\" (UniqueName: \"kubernetes.io/projected/e154ec1f-76b5-4d6a-8548-e2f43f31a754-kube-api-access-kdkbd\") pod \"test-curl-7600-c9275c\" (UID: \"e154ec1f-76b5-4d6a-8548-e2f43f31a754\") " pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:08.795430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.795399 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:08.846027 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.845982 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-703b9c" podUID="c4eb0d94-96ef-4851-af45-255650fb3f91" containerName="test-curl-7600-703b9c" containerID="cri-o://6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce" gracePeriod=2 Nov 03 14:29:08.849577 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:08.849553 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:08.894037 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:08.893999 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-703b9c\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="c4eb0d94-96ef-4851-af45-255650fb3f91" pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:09.224475 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:09.224445 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-c9275c"] Nov 03 14:29:09.226194 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:29:09.226160 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode154ec1f_76b5_4d6a_8548_e2f43f31a754.slice/crio-f3ed0d01c986d94a9b862bea8723b91c4753872cc2231255774e30141f2a53c5 WatchSource:0}: Error finding container f3ed0d01c986d94a9b862bea8723b91c4753872cc2231255774e30141f2a53c5: Status 404 returned error can't find the container with id f3ed0d01c986d94a9b862bea8723b91c4753872cc2231255774e30141f2a53c5 Nov 03 14:29:09.855466 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:09.855427 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-c9275c" event={"ID":"e154ec1f-76b5-4d6a-8548-e2f43f31a754","Type":"ContainerStarted","Data":"4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72"} Nov 03 14:29:09.855913 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:09.855479 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-c9275c" event={"ID":"e154ec1f-76b5-4d6a-8548-e2f43f31a754","Type":"ContainerStarted","Data":"f3ed0d01c986d94a9b862bea8723b91c4753872cc2231255774e30141f2a53c5"} Nov 03 14:29:09.963318 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:09.963264 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7600-c9275c" podStartSLOduration=1.963247 podStartE2EDuration="1.963247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:09.961123365 +0000 UTC m=+5083.155526958" watchObservedRunningTime="2026-11-03 14:29:09.963247 +0000 UTC m=+5083.157650594" Nov 03 14:29:10.333462 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:10.333420 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:10.479983 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:10.479948 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:11.058565 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.058538 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:11.147805 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.147725 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4eb0d94-96ef-4851-af45-255650fb3f91-kube-api-access-85x4g\" (UniqueName: \"kubernetes.io/projected/c4eb0d94-96ef-4851-af45-255650fb3f91-kube-api-access-85x4g\") pod \"c4eb0d94-96ef-4851-af45-255650fb3f91\" (UID: \"c4eb0d94-96ef-4851-af45-255650fb3f91\") " Nov 03 14:29:11.149742 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.149710 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eb0d94-96ef-4851-af45-255650fb3f91-kube-api-access-85x4g" pod "c4eb0d94-96ef-4851-af45-255650fb3f91" (UID: "c4eb0d94-96ef-4851-af45-255650fb3f91"). InnerVolumeSpecName "kube-api-access-85x4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:11.249461 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.249425 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85x4g\" (UniqueName: \"kubernetes.io/projected/c4eb0d94-96ef-4851-af45-255650fb3f91-kube-api-access-85x4g\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:11.334046 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.333986 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-c9275c"] Nov 03 14:29:11.337926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.337892 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-c9275c"] Nov 03 14:29:11.868295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.868258 4924 generic.go:356] "Generic (PLEG): container finished" podID="c4eb0d94-96ef-4851-af45-255650fb3f91" containerID="6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce" exitCode=137 Nov 03 14:29:11.868295 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.868299 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:11.868555 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.868352 4924 scope.go:122] "RemoveContainer" containerID="6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce" Nov 03 14:29:11.868615 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.868549 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7600-c9275c" podUID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" containerName="test-curl-7600-c9275c" containerID="cri-o://4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72" gracePeriod=2 Nov 03 14:29:11.878876 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.878854 4924 scope.go:122] "RemoveContainer" containerID="6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce" Nov 03 14:29:11.879164 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:11.879145 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce\": container with ID starting with 6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce not found: ID does not exist" containerID="6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce" Nov 03 14:29:11.879250 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.879172 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce"} err="failed to get container status \"6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce\": rpc error: code = NotFound desc = could not find container \"6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce\": container with ID starting with 6b64755d40e9793d3aa97d1676f6f3e830c816d71ddd317291f728db3fae77ce not found: ID does not exist" Nov 03 14:29:11.916556 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:11.916515 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-c9275c\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:11.960365 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:11.960332 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:11.964368 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:11.964340 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-703b9c\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="c4eb0d94-96ef-4851-af45-255650fb3f91" pod="opendatahub/test-curl-7600-703b9c" Nov 03 14:29:12.012436 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:12.012386 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-c9275c\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:12.950824 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:12.950793 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4eb0d94-96ef-4851-af45-255650fb3f91" path="/var/lib/kubelet/pods/c4eb0d94-96ef-4851-af45-255650fb3f91/volumes" Nov 03 14:29:13.903225 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:13.903197 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:14.114886 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.114860 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:14.180870 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.180838 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e154ec1f-76b5-4d6a-8548-e2f43f31a754-kube-api-access-kdkbd\" (UniqueName: \"kubernetes.io/projected/e154ec1f-76b5-4d6a-8548-e2f43f31a754-kube-api-access-kdkbd\") pod \"e154ec1f-76b5-4d6a-8548-e2f43f31a754\" (UID: \"e154ec1f-76b5-4d6a-8548-e2f43f31a754\") " Nov 03 14:29:14.182971 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.182946 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e154ec1f-76b5-4d6a-8548-e2f43f31a754-kube-api-access-kdkbd" pod "e154ec1f-76b5-4d6a-8548-e2f43f31a754" (UID: "e154ec1f-76b5-4d6a-8548-e2f43f31a754"). InnerVolumeSpecName "kube-api-access-kdkbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:14.282512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.282467 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdkbd\" (UniqueName: \"kubernetes.io/projected/e154ec1f-76b5-4d6a-8548-e2f43f31a754-kube-api-access-kdkbd\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:14.886102 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.886052 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:14.894033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.893997 4924 generic.go:356] "Generic (PLEG): container finished" podID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" containerID="4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72" exitCode=137 Nov 03 14:29:14.894202 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.894039 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:14.894202 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.894124 4924 scope.go:122] "RemoveContainer" containerID="4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72" Nov 03 14:29:14.904045 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.904019 4924 scope.go:122] "RemoveContainer" containerID="4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72" Nov 03 14:29:14.904438 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:14.904400 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72\": container with ID starting with 4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72 not found: ID does not exist" containerID="4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72" Nov 03 14:29:14.904587 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.904555 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72"} err="failed to get container status \"4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72\": rpc error: code = NotFound desc = could not find container \"4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72\": container with ID starting with 4084b1e7f279f037625683a480b48b75ab9b0cac5670ff648f639e6dd16a5a72 not found: ID does not exist" Nov 03 14:29:14.950952 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:14.950917 4924 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7600-c9275c\" is forbidden: User \"system:node:ip-10-0-129-63.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-63.us-west-2.compute.internal' and this object" podUID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" pod="opendatahub/test-curl-7600-c9275c" Nov 03 14:29:14.952543 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:14.952515 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" path="/var/lib/kubelet/pods/e154ec1f-76b5-4d6a-8548-e2f43f31a754/volumes" Nov 03 14:29:16.286714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:16.286672 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:16.445184 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:16.445144 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:17.813204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:17.813168 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:18.317325 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:18.317269 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:18.558201 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:18.558164 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:29:18.777654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:18.777620 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:19.606942 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:19.606884 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:21.564406 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:21.564351 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:21.715897 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:21.715833 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:23.973758 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:23.973721 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:25.073411 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:25.073342 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:25.791018 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:25.790975 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:27.077654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:27.077617 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:27.195811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:27.195768 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:27.807403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:27.807370 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:29:27.826841 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:27.826815 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:29:30.156411 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:30.156359 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:31.874823 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:31.874787 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:32.923748 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:32.923717 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:33.048884 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:33.048795 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:35.603571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:35.603530 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:37.846715 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:37.846680 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:40.060053 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:40.060016 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:41.891731 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:41.891686 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:43.928603 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:43.928566 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:45.138872 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:45.138829 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:47.111685 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:47.111650 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:47.585884 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:47.585846 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:47.753730 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:47.753687 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:48.558497 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:29:48.558450 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:29:48.905890 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:48.905805 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:49.158773 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:49.158740 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:49.624955 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:49.624860 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:49.763605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:49.763564 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:50.178419 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:50.178379 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:50.306193 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:50.306156 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:50.658189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:50.658157 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:52.786950 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:52.786913 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:55.149656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:55.149615 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2"] Nov 03 14:29:55.194060 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:55.194023 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw"] Nov 03 14:29:55.286673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:55.286638 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:56.478506 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:56.478468 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:56.627912 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:56.627874 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:29:57.985605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:29:57.985573 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:00.170899 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.170862 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj"] Nov 03 14:30:00.171437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.171306 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" containerName="test-curl-7600-c9275c" Nov 03 14:30:00.171437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.171319 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" containerName="test-curl-7600-c9275c" Nov 03 14:30:00.171437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.171431 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e154ec1f-76b5-4d6a-8548-e2f43f31a754" containerName="test-curl-7600-c9275c" Nov 03 14:30:00.174935 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.174917 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:00.261525 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.261477 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj"] Nov 03 14:30:00.298758 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.298718 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvf8c\" (UniqueName: \"kubernetes.io/projected/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-kube-api-access-jvf8c\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj\" (UID: \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:00.298933 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.298780 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj\" (UID: \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:00.326445 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.326409 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:30:00.400403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.400347 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvf8c\" (UniqueName: \"kubernetes.io/projected/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-kube-api-access-jvf8c\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj\" (UID: \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:00.400403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.400389 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj\" (UID: \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:00.401094 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.401056 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj\" (UID: \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:00.455124 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.455001 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvf8c\" (UniqueName: \"kubernetes.io/projected/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-kube-api-access-jvf8c\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj\" (UID: \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:00.494578 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.494540 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:00.864689 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:00.864655 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj"] Nov 03 14:30:00.866674 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:30:00.866639 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c95f3a8_91d3_4a5c_8f8e_d4103a9c2a4b.slice/crio-9059b8f6ee7f9709fbcec84cc86038524a85082a10e8c7fc5f54ced6b4d89e13 WatchSource:0}: Error finding container 9059b8f6ee7f9709fbcec84cc86038524a85082a10e8c7fc5f54ced6b4d89e13: Status 404 returned error can't find the container with id 9059b8f6ee7f9709fbcec84cc86038524a85082a10e8c7fc5f54ced6b4d89e13 Nov 03 14:30:01.208518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:01.208475 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" event={"ID":"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b","Type":"ContainerStarted","Data":"1457fd373ff5853b498e49982e6027d61fa0fda65d07f27583354183bbf8a97e"} Nov 03 14:30:01.209152 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:01.208541 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" event={"ID":"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b","Type":"ContainerStarted","Data":"9059b8f6ee7f9709fbcec84cc86038524a85082a10e8c7fc5f54ced6b4d89e13"} Nov 03 14:30:01.319255 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:01.319192 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" podStartSLOduration=1.319177173 podStartE2EDuration="1.319177173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.317761845 +0000 UTC m=+5134.512165440" watchObservedRunningTime="2026-11-03 14:30:01.319177173 +0000 UTC m=+5134.513580766" Nov 03 14:30:02.216009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:02.215961 4924 generic.go:356] "Generic (PLEG): container finished" podID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerID="1457fd373ff5853b498e49982e6027d61fa0fda65d07f27583354183bbf8a97e" exitCode=60 Nov 03 14:30:02.216456 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:02.216012 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" event={"ID":"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b","Type":"ContainerDied","Data":"1457fd373ff5853b498e49982e6027d61fa0fda65d07f27583354183bbf8a97e"} Nov 03 14:30:02.216456 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:02.216279 4924 scope.go:122] "RemoveContainer" containerID="1457fd373ff5853b498e49982e6027d61fa0fda65d07f27583354183bbf8a97e" Nov 03 14:30:02.677751 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:02.677694 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:02.802492 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:02.802395 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:03.225201 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:03.225166 4924 generic.go:356] "Generic (PLEG): container finished" podID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerID="7aaf9fc2697c79e1110a16179124cc3d91ff8f5950401bc56ede2025bb461213" exitCode=60 Nov 03 14:30:03.225636 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:03.225206 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" event={"ID":"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b","Type":"ContainerDied","Data":"7aaf9fc2697c79e1110a16179124cc3d91ff8f5950401bc56ede2025bb461213"} Nov 03 14:30:03.225636 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:03.225265 4924 scope.go:122] "RemoveContainer" containerID="1457fd373ff5853b498e49982e6027d61fa0fda65d07f27583354183bbf8a97e" Nov 03 14:30:03.225636 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:03.225456 4924 scope.go:122] "RemoveContainer" containerID="7aaf9fc2697c79e1110a16179124cc3d91ff8f5950401bc56ede2025bb461213" Nov 03 14:30:03.225775 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:03.225673 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj_odh-ai-gateway-infra(2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" Nov 03 14:30:04.232513 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:04.232488 4924 scope.go:122] "RemoveContainer" containerID="7aaf9fc2697c79e1110a16179124cc3d91ff8f5950401bc56ede2025bb461213" Nov 03 14:30:04.232917 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:04.232726 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj_odh-ai-gateway-infra(2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" Nov 03 14:30:04.394422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:04.394387 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:09.038533 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:09.038500 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:09.518773 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:09.518731 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:09.662490 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:09.662427 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:10.171485 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.171445 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" podUID="077a2a59-e1a7-4f11-af5d-4373cc6643ad" containerName="main" containerID="cri-o://1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c" gracePeriod=45 Nov 03 14:30:10.215095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.214126 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" containerID="cri-o://92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752" gracePeriod=45 Nov 03 14:30:10.534676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.534648 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 14:30:10.561350 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.561320 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.598964 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-model-cache\") pod \"b0b82e11-cb81-4df8-8d5e-65398626aaae\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599039 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-home\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-home\") pod \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599089 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b82e11-cb81-4df8-8d5e-65398626aaae-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b82e11-cb81-4df8-8d5e-65398626aaae-tls-certs\") pod \"b0b82e11-cb81-4df8-8d5e-65398626aaae\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599118 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-dshm\") pod \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599157 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-model-cache\") pod \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599188 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-home\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-home\") pod \"b0b82e11-cb81-4df8-8d5e-65398626aaae\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599229 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b82e11-cb81-4df8-8d5e-65398626aaae-kube-api-access-29ptk\" (UniqueName: \"kubernetes.io/projected/b0b82e11-cb81-4df8-8d5e-65398626aaae-kube-api-access-29ptk\") pod \"b0b82e11-cb81-4df8-8d5e-65398626aaae\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599329 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tmp-dir\") pod \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599367 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077a2a59-e1a7-4f11-af5d-4373cc6643ad-kube-api-access-mx8jc\" (UniqueName: \"kubernetes.io/projected/077a2a59-e1a7-4f11-af5d-4373cc6643ad-kube-api-access-mx8jc\") pod \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599418 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-tmp-dir\") pod \"b0b82e11-cb81-4df8-8d5e-65398626aaae\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599506 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tls-certs\") pod \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\" (UID: \"077a2a59-e1a7-4f11-af5d-4373cc6643ad\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.599551 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-dshm\") pod \"b0b82e11-cb81-4df8-8d5e-65398626aaae\" (UID: \"b0b82e11-cb81-4df8-8d5e-65398626aaae\") " Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.600510 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-model-cache" pod "b0b82e11-cb81-4df8-8d5e-65398626aaae" (UID: "b0b82e11-cb81-4df8-8d5e-65398626aaae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.600776 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tmp-dir" pod "077a2a59-e1a7-4f11-af5d-4373cc6643ad" (UID: "077a2a59-e1a7-4f11-af5d-4373cc6643ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.600972 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-model-cache" pod "077a2a59-e1a7-4f11-af5d-4373cc6643ad" (UID: "077a2a59-e1a7-4f11-af5d-4373cc6643ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.601159 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-home" pod "077a2a59-e1a7-4f11-af5d-4373cc6643ad" (UID: "077a2a59-e1a7-4f11-af5d-4373cc6643ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.601323 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-tmp-dir" pod "b0b82e11-cb81-4df8-8d5e-65398626aaae" (UID: "b0b82e11-cb81-4df8-8d5e-65398626aaae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.604322 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-home" pod "b0b82e11-cb81-4df8-8d5e-65398626aaae" (UID: "b0b82e11-cb81-4df8-8d5e-65398626aaae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.604939 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.604966 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.604981 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.604996 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.605010 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.606218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.605024 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.632282 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.632228 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077a2a59-e1a7-4f11-af5d-4373cc6643ad-kube-api-access-mx8jc" pod "077a2a59-e1a7-4f11-af5d-4373cc6643ad" (UID: "077a2a59-e1a7-4f11-af5d-4373cc6643ad"). InnerVolumeSpecName "kube-api-access-mx8jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:10.639494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.639456 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-dshm" pod "077a2a59-e1a7-4f11-af5d-4373cc6643ad" (UID: "077a2a59-e1a7-4f11-af5d-4373cc6643ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.639494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.639465 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tls-certs" pod "077a2a59-e1a7-4f11-af5d-4373cc6643ad" (UID: "077a2a59-e1a7-4f11-af5d-4373cc6643ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:10.639738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.639472 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b82e11-cb81-4df8-8d5e-65398626aaae-kube-api-access-29ptk" pod "b0b82e11-cb81-4df8-8d5e-65398626aaae" (UID: "b0b82e11-cb81-4df8-8d5e-65398626aaae"). InnerVolumeSpecName "kube-api-access-29ptk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:10.639738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.639602 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b82e11-cb81-4df8-8d5e-65398626aaae-tls-certs" pod "b0b82e11-cb81-4df8-8d5e-65398626aaae" (UID: "b0b82e11-cb81-4df8-8d5e-65398626aaae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:10.640986 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.640953 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-dshm" pod "b0b82e11-cb81-4df8-8d5e-65398626aaae" (UID: "b0b82e11-cb81-4df8-8d5e-65398626aaae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.706134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.706097 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b82e11-cb81-4df8-8d5e-65398626aaae-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.706134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.706134 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/077a2a59-e1a7-4f11-af5d-4373cc6643ad-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.706371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.706149 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ptk\" (UniqueName: \"kubernetes.io/projected/b0b82e11-cb81-4df8-8d5e-65398626aaae-kube-api-access-29ptk\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.706371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.706167 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx8jc\" (UniqueName: \"kubernetes.io/projected/077a2a59-e1a7-4f11-af5d-4373cc6643ad-kube-api-access-mx8jc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.706371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.706184 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/077a2a59-e1a7-4f11-af5d-4373cc6643ad-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.706371 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:10.706198 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b82e11-cb81-4df8-8d5e-65398626aaae-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:11.006857 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.006822 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:11.277363 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.277263 4924 generic.go:356] "Generic (PLEG): container finished" podID="077a2a59-e1a7-4f11-af5d-4373cc6643ad" containerID="1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c" exitCode=0 Nov 03 14:30:11.277363 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.277315 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" Nov 03 14:30:11.277903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.277351 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" event={"ID":"077a2a59-e1a7-4f11-af5d-4373cc6643ad","Type":"ContainerDied","Data":"1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c"} Nov 03 14:30:11.277903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.277399 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2" event={"ID":"077a2a59-e1a7-4f11-af5d-4373cc6643ad","Type":"ContainerDied","Data":"451f2c9813db082dafcbd5e4255d4a0321beae61cdc3aaf963bf6e201fd941df"} Nov 03 14:30:11.277903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.277419 4924 scope.go:122] "RemoveContainer" containerID="1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c" Nov 03 14:30:11.279263 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.279238 4924 generic.go:356] "Generic (PLEG): container finished" podID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerID="92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752" exitCode=0 Nov 03 14:30:11.279512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.279285 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" Nov 03 14:30:11.279512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.279323 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" event={"ID":"b0b82e11-cb81-4df8-8d5e-65398626aaae","Type":"ContainerDied","Data":"92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752"} Nov 03 14:30:11.279512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.279372 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw" event={"ID":"b0b82e11-cb81-4df8-8d5e-65398626aaae","Type":"ContainerDied","Data":"97f901397fad905a08370506346903461e0bf35e71ca81455a17360d5845107f"} Nov 03 14:30:11.287486 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.287462 4924 scope.go:122] "RemoveContainer" containerID="1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c" Nov 03 14:30:11.287775 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:11.287752 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c\": container with ID starting with 1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c not found: ID does not exist" containerID="1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c" Nov 03 14:30:11.287871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.287787 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c"} err="failed to get container status \"1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c\": rpc error: code = NotFound desc = could not find container \"1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c\": container with ID starting with 1d33b1dbb9b19c5106a96f9adb50568ad7ef8aa0cf3da03e33ac8d949caa5e3c not found: ID does not exist" Nov 03 14:30:11.287871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.287812 4924 scope.go:122] "RemoveContainer" containerID="92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752" Nov 03 14:30:11.296666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.296648 4924 scope.go:122] "RemoveContainer" containerID="92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752" Nov 03 14:30:11.297007 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:11.296980 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752\": container with ID starting with 92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752 not found: ID does not exist" containerID="92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752" Nov 03 14:30:11.297086 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.297015 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752"} err="failed to get container status \"92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752\": rpc error: code = NotFound desc = could not find container \"92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752\": container with ID starting with 92f5daea668295a11a9cdfbd8ed213d638245c607ed6abdb4a01fdbcfb716752 not found: ID does not exist" Nov 03 14:30:11.434346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.434312 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2"] Nov 03 14:30:11.436835 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.436810 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-w1-34b13c-kserve-855595cc76-vckl2"] Nov 03 14:30:11.511638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.511590 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj"] Nov 03 14:30:11.511897 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.511861 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" containerID="cri-o://767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42" gracePeriod=30 Nov 03 14:30:11.534733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.534637 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:11.541020 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.540987 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw"] Nov 03 14:30:11.544502 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:11.544478 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/simulator-w1-34b13c-kserve-8564c4df6d-h4tcw"] Nov 03 14:30:12.949793 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:12.949764 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077a2a59-e1a7-4f11-af5d-4373cc6643ad" path="/var/lib/kubelet/pods/077a2a59-e1a7-4f11-af5d-4373cc6643ad/volumes" Nov 03 14:30:12.950239 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:12.950224 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" path="/var/lib/kubelet/pods/b0b82e11-cb81-4df8-8d5e-65398626aaae/volumes" Nov 03 14:30:13.024747 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:13.024713 4924 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:30:13.024962 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:13.024775 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:30:14.666351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:14.666315 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:14.945625 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:14.945542 4924 scope.go:122] "RemoveContainer" containerID="7aaf9fc2697c79e1110a16179124cc3d91ff8f5950401bc56ede2025bb461213" Nov 03 14:30:15.307590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:15.307549 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" event={"ID":"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b","Type":"ContainerStarted","Data":"72315d4dfb2924c1ef492361d2b19ac5eb57da6636cd999a38a590f840ba6165"} Nov 03 14:30:15.313273 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:15.313239 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:16.315387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.315347 4924 generic.go:356] "Generic (PLEG): container finished" podID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerID="72315d4dfb2924c1ef492361d2b19ac5eb57da6636cd999a38a590f840ba6165" exitCode=60 Nov 03 14:30:16.315831 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.315412 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" event={"ID":"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b","Type":"ContainerDied","Data":"72315d4dfb2924c1ef492361d2b19ac5eb57da6636cd999a38a590f840ba6165"} Nov 03 14:30:16.315831 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.315471 4924 scope.go:122] "RemoveContainer" containerID="7aaf9fc2697c79e1110a16179124cc3d91ff8f5950401bc56ede2025bb461213" Nov 03 14:30:16.315926 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.315869 4924 scope.go:122] "RemoveContainer" containerID="72315d4dfb2924c1ef492361d2b19ac5eb57da6636cd999a38a590f840ba6165" Nov 03 14:30:16.316220 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:16.316190 4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj_odh-ai-gateway-infra(2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" Nov 03 14:30:16.434517 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.434484 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj"] Nov 03 14:30:16.786406 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.786379 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 14:30:16.871100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871037 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-certs\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871116 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-envoy\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871147 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-kube-api-access-qkh6c\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-kube-api-access-qkh6c\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871176 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-token\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871208 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb7e333-ae06-4712-acf2-792344bbd8eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7e333-ae06-4712-acf2-792344bbd8eb-istiod-ca-cert\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871238 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-socket\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871276 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-credential-socket\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871369 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-podinfo\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871371 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-certs" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:16.871695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871398 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-data\") pod \"fbb7e333-ae06-4712-acf2-792344bbd8eb\" (UID: \"fbb7e333-ae06-4712-acf2-792344bbd8eb\") " Nov 03 14:30:16.871695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871623 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-credential-socket" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:16.871695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871671 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb7e333-ae06-4712-acf2-792344bbd8eb-istiod-ca-cert" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:16.871975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871803 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:16.871975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871824 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7e333-ae06-4712-acf2-792344bbd8eb-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:16.871975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871839 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:16.871975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.871881 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-data" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:16.872215 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.872134 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-socket" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:16.873630 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.873603 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-token" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:16.873736 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.873669 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-podinfo" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:30:16.873891 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.873866 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-kube-api-access-qkh6c" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "kube-api-access-qkh6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:16.873972 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.873894 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-envoy" pod "fbb7e333-ae06-4712-acf2-792344bbd8eb" (UID: "fbb7e333-ae06-4712-acf2-792344bbd8eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:16.973171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.973138 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:16.973291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.973173 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkh6c\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-kube-api-access-qkh6c\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:16.973291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.973188 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:16.973291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.973201 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:16.973291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.973215 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:16.973291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:16.973242 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7e333-ae06-4712-acf2-792344bbd8eb-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.322743 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.322653 4924 generic.go:356] "Generic (PLEG): container finished" podID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerID="767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42" exitCode=0 Nov 03 14:30:17.322743 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.322706 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" Nov 03 14:30:17.323323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.322738 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" event={"ID":"fbb7e333-ae06-4712-acf2-792344bbd8eb","Type":"ContainerDied","Data":"767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42"} Nov 03 14:30:17.323323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.322797 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj" event={"ID":"fbb7e333-ae06-4712-acf2-792344bbd8eb","Type":"ContainerDied","Data":"a6a42404470900097319c6bf14b0debfc649ca99a9fd48b73c2ee7f1f87f6af2"} Nov 03 14:30:17.323323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.322828 4924 scope.go:122] "RemoveContainer" containerID="767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42" Nov 03 14:30:17.333016 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.332993 4924 scope.go:122] "RemoveContainer" containerID="767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42" Nov 03 14:30:17.333346 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:17.333322 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42\": container with ID starting with 767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42 not found: ID does not exist" containerID="767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42" Nov 03 14:30:17.333443 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.333354 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42"} err="failed to get container status \"767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42\": rpc error: code = NotFound desc = could not find container \"767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42\": container with ID starting with 767ddc599aa4ec31c965ec33adb7002e3f88aca05b007ef6e5a427cb61279f42 not found: ID does not exist" Nov 03 14:30:17.457186 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.457162 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:17.488361 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.488333 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj"] Nov 03 14:30:17.535811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.535775 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-xfsnj"] Nov 03 14:30:17.578007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.577917 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-maas-api-service-ca\") pod \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\" (UID: \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\") " Nov 03 14:30:17.578007 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.577998 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-kube-api-access-jvf8c\" (UniqueName: \"kubernetes.io/projected/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-kube-api-access-jvf8c\") pod \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\" (UID: \"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b\") " Nov 03 14:30:17.578384 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.578359 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-maas-api-service-ca" pod "2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" (UID: "2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:17.580188 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.580155 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-kube-api-access-jvf8c" pod "2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" (UID: "2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b"). InnerVolumeSpecName "kube-api-access-jvf8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:17.679661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.679620 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvf8c\" (UniqueName: \"kubernetes.io/projected/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-kube-api-access-jvf8c\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.679661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:17.679659 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b-maas-api-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:18.334807 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:18.334765 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" event={"ID":"2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b","Type":"ContainerDied","Data":"9059b8f6ee7f9709fbcec84cc86038524a85082a10e8c7fc5f54ced6b4d89e13"} Nov 03 14:30:18.335300 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:18.334816 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj" Nov 03 14:30:18.335300 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:18.334846 4924 scope.go:122] "RemoveContainer" containerID="72315d4dfb2924c1ef492361d2b19ac5eb57da6636cd999a38a590f840ba6165" Nov 03 14:30:18.497183 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:18.497152 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj"] Nov 03 14:30:18.545572 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:18.545537 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895270-c5xqj"] Nov 03 14:30:18.567235 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:18.567181 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:30:18.950137 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:18.950098 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" path="/var/lib/kubelet/pods/2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b/volumes" Nov 03 14:30:18.950703 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:18.950681 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" path="/var/lib/kubelet/pods/fbb7e333-ae06-4712-acf2-792344bbd8eb/volumes" Nov 03 14:30:19.456485 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:19.456442 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5"] Nov 03 14:30:19.474245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:19.474210 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:20.751576 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:20.751537 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:25.984355 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:25.984313 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:29.764495 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:29.764455 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:31.112224 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:31.112190 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:32.007005 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:32.006961 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:32.133486 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:32.133448 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:33.190702 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:33.190668 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:33.876339 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:33.876301 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w"] Nov 03 14:30:33.889390 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:33.889357 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:34.447992 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.447962 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:34.476709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.476663 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" podUID="d47013a3-ed77-44a3-917b-b05d224cd6f5" containerName="main" containerID="cri-o://7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5" gracePeriod=45 Nov 03 14:30:34.733419 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.733390 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:30:34.865303 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.865263 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-model-cache\") pod \"d47013a3-ed77-44a3-917b-b05d224cd6f5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " Nov 03 14:30:34.865577 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.865337 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-dshm\") pod \"d47013a3-ed77-44a3-917b-b05d224cd6f5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " Nov 03 14:30:34.865577 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.865373 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47013a3-ed77-44a3-917b-b05d224cd6f5-kube-api-access-n6h8z\" (UniqueName: \"kubernetes.io/projected/d47013a3-ed77-44a3-917b-b05d224cd6f5-kube-api-access-n6h8z\") pod \"d47013a3-ed77-44a3-917b-b05d224cd6f5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " Nov 03 14:30:34.865577 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.865403 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-home\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-home\") pod \"d47013a3-ed77-44a3-917b-b05d224cd6f5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " Nov 03 14:30:34.866055 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.866013 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-tmp-dir\") pod \"d47013a3-ed77-44a3-917b-b05d224cd6f5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " Nov 03 14:30:34.866204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.866060 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-home" pod "d47013a3-ed77-44a3-917b-b05d224cd6f5" (UID: "d47013a3-ed77-44a3-917b-b05d224cd6f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:34.866204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.866117 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47013a3-ed77-44a3-917b-b05d224cd6f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d47013a3-ed77-44a3-917b-b05d224cd6f5-tls-certs\") pod \"d47013a3-ed77-44a3-917b-b05d224cd6f5\" (UID: \"d47013a3-ed77-44a3-917b-b05d224cd6f5\") " Nov 03 14:30:34.866558 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.866537 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:34.866658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.866588 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-tmp-dir" pod "d47013a3-ed77-44a3-917b-b05d224cd6f5" (UID: "d47013a3-ed77-44a3-917b-b05d224cd6f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:34.866658 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.866611 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-model-cache" pod "d47013a3-ed77-44a3-917b-b05d224cd6f5" (UID: "d47013a3-ed77-44a3-917b-b05d224cd6f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:34.872104 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.869317 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-dshm" pod "d47013a3-ed77-44a3-917b-b05d224cd6f5" (UID: "d47013a3-ed77-44a3-917b-b05d224cd6f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:34.872104 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.869367 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47013a3-ed77-44a3-917b-b05d224cd6f5-tls-certs" pod "d47013a3-ed77-44a3-917b-b05d224cd6f5" (UID: "d47013a3-ed77-44a3-917b-b05d224cd6f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:34.872104 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.870447 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47013a3-ed77-44a3-917b-b05d224cd6f5-kube-api-access-n6h8z" pod "d47013a3-ed77-44a3-917b-b05d224cd6f5" (UID: "d47013a3-ed77-44a3-917b-b05d224cd6f5"). InnerVolumeSpecName "kube-api-access-n6h8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:34.967601 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.967528 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47013a3-ed77-44a3-917b-b05d224cd6f5-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:34.967601 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.967556 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:34.967601 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.967570 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:34.967601 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.967583 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6h8z\" (UniqueName: \"kubernetes.io/projected/d47013a3-ed77-44a3-917b-b05d224cd6f5-kube-api-access-n6h8z\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:34.967601 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:34.967593 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47013a3-ed77-44a3-917b-b05d224cd6f5-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:35.440004 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.439969 4924 generic.go:356] "Generic (PLEG): container finished" podID="d47013a3-ed77-44a3-917b-b05d224cd6f5" containerID="7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5" exitCode=0 Nov 03 14:30:35.440197 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.440028 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" Nov 03 14:30:35.440197 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.440024 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" event={"ID":"d47013a3-ed77-44a3-917b-b05d224cd6f5","Type":"ContainerDied","Data":"7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5"} Nov 03 14:30:35.440197 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.440119 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5" event={"ID":"d47013a3-ed77-44a3-917b-b05d224cd6f5","Type":"ContainerDied","Data":"b9a8d08f346020fe789699c63712569e3998ce0036c78e5223fc65f1c1c6d9a4"} Nov 03 14:30:35.440197 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.440144 4924 scope.go:122] "RemoveContainer" containerID="7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5" Nov 03 14:30:35.451026 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.451005 4924 scope.go:122] "RemoveContainer" containerID="7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5" Nov 03 14:30:35.451396 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:35.451355 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5\": container with ID starting with 7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5 not found: ID does not exist" containerID="7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5" Nov 03 14:30:35.451396 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.451383 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5"} err="failed to get container status \"7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5\": rpc error: code = NotFound desc = could not find container \"7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5\": container with ID starting with 7b69c48aaf6cd78655155e9265b30c4845331d6eebbe0c3f2402f6570800eef5 not found: ID does not exist" Nov 03 14:30:35.594234 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.594195 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5"] Nov 03 14:30:35.640779 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.640740 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/sub-test-model-2bceb1-kserve-7f6b6bcd87-8rjz5"] Nov 03 14:30:35.899346 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:35.899244 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:36.178767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:36.178733 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:36.950803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:36.950727 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47013a3-ed77-44a3-917b-b05d224cd6f5" path="/var/lib/kubelet/pods/d47013a3-ed77-44a3-917b-b05d224cd6f5/volumes" Nov 03 14:30:38.859854 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:38.859805 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:40.864357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:40.864315 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:41.441333 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:41.441295 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:42.287943 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:42.287910 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:42.429771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:42.429730 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:45.470472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:45.470422 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:46.132340 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.132304 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6"] Nov 03 14:30:46.132583 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.132555 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" podUID="9839ceb3-a0c7-4748-9f6b-307edd6bf097" containerName="maas-api" containerID="cri-o://d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4" gracePeriod=30 Nov 03 14:30:46.187001 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.186966 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:46.416842 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.416808 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 14:30:46.470383 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.470347 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-metrics-certs\") pod \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " Nov 03 14:30:46.470565 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.470524 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls\") pod \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " Nov 03 14:30:46.470935 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.470566 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9839ceb3-a0c7-4748-9f6b-307edd6bf097-kube-api-access-5bljd\" (UniqueName: \"kubernetes.io/projected/9839ceb3-a0c7-4748-9f6b-307edd6bf097-kube-api-access-5bljd\") pod \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " Nov 03 14:30:46.470935 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.470607 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9839ceb3-a0c7-4748-9f6b-307edd6bf097-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9839ceb3-a0c7-4748-9f6b-307edd6bf097-openshift-service-ca\") pod \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\" (UID: \"9839ceb3-a0c7-4748-9f6b-307edd6bf097\") " Nov 03 14:30:46.471280 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.471244 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9839ceb3-a0c7-4748-9f6b-307edd6bf097-openshift-service-ca" pod "9839ceb3-a0c7-4748-9f6b-307edd6bf097" (UID: "9839ceb3-a0c7-4748-9f6b-307edd6bf097"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:46.472770 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.472742 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9839ceb3-a0c7-4748-9f6b-307edd6bf097-kube-api-access-5bljd" pod "9839ceb3-a0c7-4748-9f6b-307edd6bf097" (UID: "9839ceb3-a0c7-4748-9f6b-307edd6bf097"). InnerVolumeSpecName "kube-api-access-5bljd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:46.472894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.472868 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls" pod "9839ceb3-a0c7-4748-9f6b-307edd6bf097" (UID: "9839ceb3-a0c7-4748-9f6b-307edd6bf097"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:46.472945 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.472879 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-metrics-certs" pod "9839ceb3-a0c7-4748-9f6b-307edd6bf097" (UID: "9839ceb3-a0c7-4748-9f6b-307edd6bf097"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:46.504175 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.504141 4924 generic.go:356] "Generic (PLEG): container finished" podID="9839ceb3-a0c7-4748-9f6b-307edd6bf097" containerID="d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4" exitCode=0 Nov 03 14:30:46.504334 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.504187 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" Nov 03 14:30:46.504334 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.504226 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" event={"ID":"9839ceb3-a0c7-4748-9f6b-307edd6bf097","Type":"ContainerDied","Data":"d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4"} Nov 03 14:30:46.504334 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.504279 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6" event={"ID":"9839ceb3-a0c7-4748-9f6b-307edd6bf097","Type":"ContainerDied","Data":"56b0682609f8093f43a0b6b282a5e733f094bb17420d718771f77e55611cde55"} Nov 03 14:30:46.504334 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.504300 4924 scope.go:122] "RemoveContainer" containerID="d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4" Nov 03 14:30:46.514025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.514006 4924 scope.go:122] "RemoveContainer" containerID="d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4" Nov 03 14:30:46.514333 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:46.514311 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4\": container with ID starting with d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4 not found: ID does not exist" containerID="d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4" Nov 03 14:30:46.514419 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.514342 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4"} err="failed to get container status \"d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4\": rpc error: code = NotFound desc = could not find container \"d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4\": container with ID starting with d26ce5e051e8deaa5abf56218a8e7c643c9d41508e87fad675ed1a687c3dd4b4 not found: ID does not exist" Nov 03 14:30:46.572179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.572144 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:46.572179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.572173 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bljd\" (UniqueName: \"kubernetes.io/projected/9839ceb3-a0c7-4748-9f6b-307edd6bf097-kube-api-access-5bljd\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:46.572179 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.572185 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9839ceb3-a0c7-4748-9f6b-307edd6bf097-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:46.572416 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.572193 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9839ceb3-a0c7-4748-9f6b-307edd6bf097-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:46.635087 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.635044 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6"] Nov 03 14:30:46.641033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.640930 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-34b13c-574d86566f-c9nk6"] Nov 03 14:30:46.950744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:46.950711 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9839ceb3-a0c7-4748-9f6b-307edd6bf097" path="/var/lib/kubelet/pods/9839ceb3-a0c7-4748-9f6b-307edd6bf097/volumes" Nov 03 14:30:47.417478 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:47.417440 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:48.102181 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:48.102145 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:48.559459 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:48.559420 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:30:48.897494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:48.897408 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" podUID="b26d3697-b507-4e1b-9add-ceca7dac937a" containerName="main" containerID="cri-o://e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff" gracePeriod=45 Nov 03 14:30:49.011955 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.011920 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" podUID="b26d3697-b507-4e1b-9add-ceca7dac937a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.183:8000/ready\": dial tcp 10.132.0.183:8000: connect: connection refused" Nov 03 14:30:49.156378 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.156294 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:49.157746 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.157727 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:30:49.195550 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.195515 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-dshm\") pod \"b26d3697-b507-4e1b-9add-ceca7dac937a\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " Nov 03 14:30:49.195735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.195562 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-home\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-home\") pod \"b26d3697-b507-4e1b-9add-ceca7dac937a\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " Nov 03 14:30:49.195821 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.195757 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26d3697-b507-4e1b-9add-ceca7dac937a-kube-api-access-qtmwg\" (UniqueName: \"kubernetes.io/projected/b26d3697-b507-4e1b-9add-ceca7dac937a-kube-api-access-qtmwg\") pod \"b26d3697-b507-4e1b-9add-ceca7dac937a\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " Nov 03 14:30:49.195821 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.195813 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26d3697-b507-4e1b-9add-ceca7dac937a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d3697-b507-4e1b-9add-ceca7dac937a-tls-certs\") pod \"b26d3697-b507-4e1b-9add-ceca7dac937a\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " Nov 03 14:30:49.195942 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.195873 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-tmp-dir\") pod \"b26d3697-b507-4e1b-9add-ceca7dac937a\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " Nov 03 14:30:49.195942 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.195898 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-home" pod "b26d3697-b507-4e1b-9add-ceca7dac937a" (UID: "b26d3697-b507-4e1b-9add-ceca7dac937a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:49.196160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.195942 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-model-cache\") pod \"b26d3697-b507-4e1b-9add-ceca7dac937a\" (UID: \"b26d3697-b507-4e1b-9add-ceca7dac937a\") " Nov 03 14:30:49.196265 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.196243 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:49.196494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.196475 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-model-cache" pod "b26d3697-b507-4e1b-9add-ceca7dac937a" (UID: "b26d3697-b507-4e1b-9add-ceca7dac937a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:49.196671 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.196651 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-tmp-dir" pod "b26d3697-b507-4e1b-9add-ceca7dac937a" (UID: "b26d3697-b507-4e1b-9add-ceca7dac937a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:49.198062 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.198030 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26d3697-b507-4e1b-9add-ceca7dac937a-tls-certs" pod "b26d3697-b507-4e1b-9add-ceca7dac937a" (UID: "b26d3697-b507-4e1b-9add-ceca7dac937a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:49.198307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.198287 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-dshm" pod "b26d3697-b507-4e1b-9add-ceca7dac937a" (UID: "b26d3697-b507-4e1b-9add-ceca7dac937a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:49.198376 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.198302 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26d3697-b507-4e1b-9add-ceca7dac937a-kube-api-access-qtmwg" pod "b26d3697-b507-4e1b-9add-ceca7dac937a" (UID: "b26d3697-b507-4e1b-9add-ceca7dac937a"). InnerVolumeSpecName "kube-api-access-qtmwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:49.296925 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.296887 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:49.296925 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.296918 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtmwg\" (UniqueName: \"kubernetes.io/projected/b26d3697-b507-4e1b-9add-ceca7dac937a-kube-api-access-qtmwg\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:49.296925 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.296929 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b26d3697-b507-4e1b-9add-ceca7dac937a-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:49.297278 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.296940 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:49.297278 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.296949 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b26d3697-b507-4e1b-9add-ceca7dac937a-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:49.298465 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.298429 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:49.522896 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.522860 4924 generic.go:356] "Generic (PLEG): container finished" podID="b26d3697-b507-4e1b-9add-ceca7dac937a" containerID="e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff" exitCode=0 Nov 03 14:30:49.523134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.522904 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" Nov 03 14:30:49.523134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.522947 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" event={"ID":"b26d3697-b507-4e1b-9add-ceca7dac937a","Type":"ContainerDied","Data":"e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff"} Nov 03 14:30:49.523134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.522994 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w" event={"ID":"b26d3697-b507-4e1b-9add-ceca7dac937a","Type":"ContainerDied","Data":"31ea555b9734c35f61a6938a6ddd64be8a3d62b2233c2a6c388211390c6fec06"} Nov 03 14:30:49.523134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.523016 4924 scope.go:122] "RemoveContainer" containerID="e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff" Nov 03 14:30:49.534366 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.534342 4924 scope.go:122] "RemoveContainer" containerID="e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff" Nov 03 14:30:49.534677 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:30:49.534656 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff\": container with ID starting with e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff not found: ID does not exist" containerID="e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff" Nov 03 14:30:49.534767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.534687 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff"} err="failed to get container status \"e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff\": rpc error: code = NotFound desc = could not find container \"e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff\": container with ID starting with e3c081e1830b922dcf4218d10aac6d2544bac16cfbc0b43e7b0ffa85c182f1ff not found: ID does not exist" Nov 03 14:30:49.679650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.679618 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w"] Nov 03 14:30:49.726190 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:49.726153 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/sub-test-model-d341af-kserve-6c4cc7659f-67b2w"] Nov 03 14:30:50.957001 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:50.956967 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26d3697-b507-4e1b-9add-ceca7dac937a" path="/var/lib/kubelet/pods/b26d3697-b507-4e1b-9add-ceca7dac937a/volumes" Nov 03 14:30:53.116765 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:53.116729 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:53.256673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:53.256630 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:53.844955 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:53.844908 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:53.975870 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:53.975835 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:54.852045 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:54.852006 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:55.338771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:55.338735 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:30:55.344154 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:55.344128 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2"] Nov 03 14:30:55.344364 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:55.344342 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" containerID="cri-o://e6e51b42a37437b2514e2187824e01d89370407bcd41bb70ce0bae106cf68896" gracePeriod=30 Nov 03 14:30:55.772644 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:55.772600 4924 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:30:55.772809 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:55.772680 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:30:58.747852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:30:58.747816 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:00.589673 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.589643 4924 generic.go:356] "Generic (PLEG): container finished" podID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerID="e6e51b42a37437b2514e2187824e01d89370407bcd41bb70ce0bae106cf68896" exitCode=0 Nov 03 14:31:00.590008 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.589713 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" event={"ID":"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e","Type":"ContainerDied","Data":"e6e51b42a37437b2514e2187824e01d89370407bcd41bb70ce0bae106cf68896"} Nov 03 14:31:00.607138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.607053 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 14:31:00.708758 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.708717 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-data\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.708946 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.708773 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-token\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-token\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.708946 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.708803 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-socket\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.708946 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.708849 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-kube-api-access-jvqcc\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-kube-api-access-jvqcc\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.708946 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.708878 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-certs\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.709236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709018 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-envoy\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.709236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709037 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-data" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.709236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709103 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istiod-ca-cert\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.709236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709189 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-certs" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.709236 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709194 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-credential-socket\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.709494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709267 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-podinfo\") pod \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\" (UID: \"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e\") " Nov 03 14:31:00.709494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709295 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-socket" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.709494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709413 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-credential-socket" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.709647 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709532 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istiod-ca-cert" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:00.709735 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709714 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.709806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709741 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.709806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709757 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.709806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709770 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.709806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.709784 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.711276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.711249 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-token" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:00.711381 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.711346 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-podinfo" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:31:00.711529 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.711505 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-envoy" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.711626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.711525 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-kube-api-access-jvqcc" pod "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" (UID: "d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e"). InnerVolumeSpecName "kube-api-access-jvqcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:00.810281 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.810244 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqcc\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-kube-api-access-jvqcc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.810281 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.810275 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.810281 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.810286 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.810626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:00.810295 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:01.596250 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:01.596217 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" Nov 03 14:31:01.596709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:01.596222 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2" event={"ID":"d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e","Type":"ContainerDied","Data":"c653b2f938377e113dbee2d0f591ada6ccfda43602bda6e97403cbd7c049ad6e"} Nov 03 14:31:01.596709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:01.596348 4924 scope.go:122] "RemoveContainer" containerID="e6e51b42a37437b2514e2187824e01d89370407bcd41bb70ce0bae106cf68896" Nov 03 14:31:01.753780 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:01.753737 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2"] Nov 03 14:31:01.798998 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:01.798955 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-r4pl2"] Nov 03 14:31:02.949406 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:02.949373 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" path="/var/lib/kubelet/pods/d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e/volumes" Nov 03 14:31:03.272181 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.272101 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz"] Nov 03 14:31:03.272351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.272294 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" podUID="a2ff7954-e696-4dcf-90f9-ee949cd35c27" containerName="maas-api" containerID="cri-o://1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a" gracePeriod=30 Nov 03 14:31:03.363084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.363045 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:03.537109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.537028 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 14:31:03.609638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.609602 4924 generic.go:356] "Generic (PLEG): container finished" podID="a2ff7954-e696-4dcf-90f9-ee949cd35c27" containerID="1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a" exitCode=0 Nov 03 14:31:03.609806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.609650 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" Nov 03 14:31:03.609806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.609692 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" event={"ID":"a2ff7954-e696-4dcf-90f9-ee949cd35c27","Type":"ContainerDied","Data":"1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a"} Nov 03 14:31:03.609806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.609748 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz" event={"ID":"a2ff7954-e696-4dcf-90f9-ee949cd35c27","Type":"ContainerDied","Data":"bdbf593666f4846279c60fcc410359e4ea43aad12aecbc4e5ce320b4254b4866"} Nov 03 14:31:03.609806 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.609776 4924 scope.go:122] "RemoveContainer" containerID="1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a" Nov 03 14:31:03.621792 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.621770 4924 scope.go:122] "RemoveContainer" containerID="1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a" Nov 03 14:31:03.622116 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:31:03.622094 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a\": container with ID starting with 1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a not found: ID does not exist" containerID="1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a" Nov 03 14:31:03.622197 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.622125 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a"} err="failed to get container status \"1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a\": rpc error: code = NotFound desc = could not find container \"1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a\": container with ID starting with 1053d1d5a504997787919be7548e59cf8dc298c094c9fa0c38c1fb6bc01c620a not found: ID does not exist" Nov 03 14:31:03.636267 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.636243 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls\") pod \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " Nov 03 14:31:03.636364 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.636298 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ff7954-e696-4dcf-90f9-ee949cd35c27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7954-e696-4dcf-90f9-ee949cd35c27-openshift-service-ca\") pod \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " Nov 03 14:31:03.636420 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.636362 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ff7954-e696-4dcf-90f9-ee949cd35c27-kube-api-access-4mn2q\" (UniqueName: \"kubernetes.io/projected/a2ff7954-e696-4dcf-90f9-ee949cd35c27-kube-api-access-4mn2q\") pod \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " Nov 03 14:31:03.636420 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.636386 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-metrics-certs\") pod \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\" (UID: \"a2ff7954-e696-4dcf-90f9-ee949cd35c27\") " Nov 03 14:31:03.636731 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.636693 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ff7954-e696-4dcf-90f9-ee949cd35c27-openshift-service-ca" pod "a2ff7954-e696-4dcf-90f9-ee949cd35c27" (UID: "a2ff7954-e696-4dcf-90f9-ee949cd35c27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:03.638345 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.638324 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls" pod "a2ff7954-e696-4dcf-90f9-ee949cd35c27" (UID: "a2ff7954-e696-4dcf-90f9-ee949cd35c27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:03.638433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.638358 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ff7954-e696-4dcf-90f9-ee949cd35c27-kube-api-access-4mn2q" pod "a2ff7954-e696-4dcf-90f9-ee949cd35c27" (UID: "a2ff7954-e696-4dcf-90f9-ee949cd35c27"). InnerVolumeSpecName "kube-api-access-4mn2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:03.638433 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.638371 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-metrics-certs" pod "a2ff7954-e696-4dcf-90f9-ee949cd35c27" (UID: "a2ff7954-e696-4dcf-90f9-ee949cd35c27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:03.737586 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.737547 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-maas-api-tls\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:03.737586 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.737581 4924 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ff7954-e696-4dcf-90f9-ee949cd35c27-openshift-service-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:03.737586 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.737590 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mn2q\" (UniqueName: \"kubernetes.io/projected/a2ff7954-e696-4dcf-90f9-ee949cd35c27-kube-api-access-4mn2q\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:03.737884 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:03.737601 4924 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2ff7954-e696-4dcf-90f9-ee949cd35c27-metrics-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:04.024204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:04.024170 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz"] Nov 03 14:31:04.027772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:04.027740 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bceb1-6947c9c566-rlnmz"] Nov 03 14:31:04.950669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:04.950633 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ff7954-e696-4dcf-90f9-ee949cd35c27" path="/var/lib/kubelet/pods/a2ff7954-e696-4dcf-90f9-ee949cd35c27/volumes" Nov 03 14:31:05.156466 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:05.156419 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:05.967348 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:05.967311 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:06.099873 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:06.099842 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:10.269426 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:10.269390 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:10.803985 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:10.803949 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:10.928395 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:10.928363 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:12.599533 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:12.599496 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:13.941836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:13.941797 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:13.943686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:13.943663 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz"] Nov 03 14:31:13.943892 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:13.943855 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" containerID="cri-o://f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96" gracePeriod=30 Nov 03 14:31:15.249955 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:15.249921 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:15.954894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:15.954862 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:18.558402 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:31:18.558358 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:31:19.198033 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.198008 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 14:31:19.384307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384220 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-socket\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384263 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-podinfo\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384307 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384284 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-token\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384314 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-credential-socket\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384362 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-envoy\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384431 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-kube-api-access-8jktr\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-kube-api-access-8jktr\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384477 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-certs\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384507 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-socket" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:19.384643 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384529 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-data\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384970 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384675 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa8e6e7-629c-4479-86ab-e810221ef398-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e6e7-629c-4479-86ab-e810221ef398-istiod-ca-cert\") pod \"ffa8e6e7-629c-4479-86ab-e810221ef398\" (UID: \"ffa8e6e7-629c-4479-86ab-e810221ef398\") " Nov 03 14:31:19.384970 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384680 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-credential-socket" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:19.384970 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.384918 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-certs" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:19.385162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.385064 4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.385162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.385097 4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-credential-socket\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.385162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.385114 4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-workload-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.385162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.385131 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-data" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:19.385363 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.385048 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa8e6e7-629c-4479-86ab-e810221ef398-istiod-ca-cert" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:19.386842 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.386811 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-kube-api-access-8jktr" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "kube-api-access-8jktr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:19.387015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.386991 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-podinfo" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:31:19.387140 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.387106 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-token" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:19.387311 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.387294 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-envoy" pod "ffa8e6e7-629c-4479-86ab-e810221ef398" (UID: "ffa8e6e7-629c-4479-86ab-e810221ef398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:19.486198 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.486162 4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-envoy\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.486198 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.486193 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jktr\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-kube-api-access-8jktr\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.486198 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.486203 4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-data\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.486455 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.486214 4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e6e7-629c-4479-86ab-e810221ef398-istiod-ca-cert\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.486455 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.486224 4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-podinfo\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.486455 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.486232 4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e6e7-629c-4479-86ab-e810221ef398-istio-token\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:19.710046 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.710005 4924 generic.go:356] "Generic (PLEG): container finished" podID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerID="f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96" exitCode=0 Nov 03 14:31:19.710568 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.710062 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" Nov 03 14:31:19.710568 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.710096 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" event={"ID":"ffa8e6e7-629c-4479-86ab-e810221ef398","Type":"ContainerDied","Data":"f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96"} Nov 03 14:31:19.710568 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.710152 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz" event={"ID":"ffa8e6e7-629c-4479-86ab-e810221ef398","Type":"ContainerDied","Data":"a3d8ffe7c6ff1d5d77df88b6b92b1b7e4a17a364e165bf53e917898d6020d7a8"} Nov 03 14:31:19.710568 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.710174 4924 scope.go:122] "RemoveContainer" containerID="f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96" Nov 03 14:31:19.720264 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.720247 4924 scope.go:122] "RemoveContainer" containerID="029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215" Nov 03 14:31:19.729013 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.728995 4924 scope.go:122] "RemoveContainer" containerID="f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96" Nov 03 14:31:19.729351 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:31:19.729332 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96\": container with ID starting with f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96 not found: ID does not exist" containerID="f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96" Nov 03 14:31:19.729436 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.729363 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96"} err="failed to get container status \"f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96\": rpc error: code = NotFound desc = could not find container \"f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96\": container with ID starting with f2aa48237fd32aaf7aa80ac602431862a301611044ad6697f393d300d1a4fa96 not found: ID does not exist" Nov 03 14:31:19.729436 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.729386 4924 scope.go:122] "RemoveContainer" containerID="029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215" Nov 03 14:31:19.729667 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:31:19.729647 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215\": container with ID starting with 029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215 not found: ID does not exist" containerID="029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215" Nov 03 14:31:19.729706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.729673 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215"} err="failed to get container status \"029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215\": rpc error: code = NotFound desc = could not find container \"029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215\": container with ID starting with 029ec680580423ef0509b14cbc8c39945397e240e27076628fb38c29ab607215 not found: ID does not exist" Nov 03 14:31:19.865427 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.865391 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz"] Nov 03 14:31:19.868128 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:19.868106 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bceb1-openshift-default-6df6464d7c-gdwbz"] Nov 03 14:31:20.949917 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:20.949884 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" path="/var/lib/kubelet/pods/ffa8e6e7-629c-4479-86ab-e810221ef398/volumes" Nov 03 14:31:21.162770 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:21.162735 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:21.285738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:21.285651 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:22.415387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:22.415357 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:22.532803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:22.532773 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:26.475493 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:26.475462 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:27.148626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:27.148590 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:31.098115 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:31.098059 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:31.576786 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:31.576750 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:36.057385 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:36.057344 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:37.865761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:37.865729 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:41.828976 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:41.828938 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:41.989240 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:41.989205 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:43.070117 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:43.070082 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:43.536723 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:43.536680 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:44.051845 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:44.051807 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:44.523881 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:44.523836 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:45.815523 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:45.815490 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:48.558998 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:31:48.558959 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:31:48.808004 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:48.807967 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:49.500031 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:49.499996 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc"] Nov 03 14:31:49.546344 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:49.546310 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2"] Nov 03 14:31:49.633004 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:49.632963 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:51.546291 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:51.546260 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:53.877200 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:53.877140 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:57.569009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:57.568977 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:57.694097 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:57.694049 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:31:57.813304 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:31:57.813266 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:04.064326 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.064294 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:04.519910 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.519868 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" podUID="a983cf0a-e87a-4615-a048-df14fb6c2f45" containerName="main" containerID="cri-o://e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b" gracePeriod=45 Nov 03 14:32:04.565876 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.565816 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" podUID="723ac416-02d4-48a5-a50b-21fb78a3fce5" containerName="main" containerID="cri-o://3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a" gracePeriod=45 Nov 03 14:32:04.825676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.825648 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 14:32:04.847699 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.847675 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 14:32:04.956582 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.956546 4924 generic.go:356] "Generic (PLEG): container finished" podID="723ac416-02d4-48a5-a50b-21fb78a3fce5" containerID="3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a" exitCode=0 Nov 03 14:32:04.956777 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.956660 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" Nov 03 14:32:04.956777 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.956684 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" event={"ID":"723ac416-02d4-48a5-a50b-21fb78a3fce5","Type":"ContainerDied","Data":"3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a"} Nov 03 14:32:04.956777 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.956725 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2" event={"ID":"723ac416-02d4-48a5-a50b-21fb78a3fce5","Type":"ContainerDied","Data":"a4919900c14c3ea7cd3666ad792ba7290f4c52795cc8ad62daf7cc615da7bb41"} Nov 03 14:32:04.956777 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.956748 4924 scope.go:122] "RemoveContainer" containerID="3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a" Nov 03 14:32:04.958651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.958569 4924 generic.go:356] "Generic (PLEG): container finished" podID="a983cf0a-e87a-4615-a048-df14fb6c2f45" containerID="e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b" exitCode=0 Nov 03 14:32:04.958651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.958610 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" event={"ID":"a983cf0a-e87a-4615-a048-df14fb6c2f45","Type":"ContainerDied","Data":"e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b"} Nov 03 14:32:04.958651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.958623 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" Nov 03 14:32:04.958651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.958632 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc" event={"ID":"a983cf0a-e87a-4615-a048-df14fb6c2f45","Type":"ContainerDied","Data":"a1c5321c802c4cac4f8f63ffab7d668aa8bc42c4ce39ce31d115e9742946ef54"} Nov 03 14:32:04.968573 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.968547 4924 scope.go:122] "RemoveContainer" containerID="3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a" Nov 03 14:32:04.968915 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:32:04.968895 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a\": container with ID starting with 3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a not found: ID does not exist" containerID="3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a" Nov 03 14:32:04.969016 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.968932 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a"} err="failed to get container status \"3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a\": rpc error: code = NotFound desc = could not find container \"3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a\": container with ID starting with 3aedc0f6bfb3def33d79dfccd223cd157deac75e7ee7b958b5acfc304770f44a not found: ID does not exist" Nov 03 14:32:04.969016 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.968951 4924 scope.go:122] "RemoveContainer" containerID="e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b" Nov 03 14:32:04.979448 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.979419 4924 scope.go:122] "RemoveContainer" containerID="e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b" Nov 03 14:32:04.979869 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:32:04.979798 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b\": container with ID starting with e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b not found: ID does not exist" containerID="e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b" Nov 03 14:32:04.979869 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.979842 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b"} err="failed to get container status \"e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b\": rpc error: code = NotFound desc = could not find container \"e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b\": container with ID starting with e776741b6dfdd44345c9b09e148df8f38efc677b779a47657a10479d90c3f37b not found: ID does not exist" Nov 03 14:32:04.985108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985083 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-home\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-home\") pod \"a983cf0a-e87a-4615-a048-df14fb6c2f45\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " Nov 03 14:32:04.985227 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985119 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a983cf0a-e87a-4615-a048-df14fb6c2f45-kube-api-access-9bjl4\" (UniqueName: \"kubernetes.io/projected/a983cf0a-e87a-4615-a048-df14fb6c2f45-kube-api-access-9bjl4\") pod \"a983cf0a-e87a-4615-a048-df14fb6c2f45\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " Nov 03 14:32:04.985227 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985156 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-dshm\") pod \"a983cf0a-e87a-4615-a048-df14fb6c2f45\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " Nov 03 14:32:04.985227 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985178 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-model-cache\") pod \"a983cf0a-e87a-4615-a048-df14fb6c2f45\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " Nov 03 14:32:04.985405 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985375 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-home" pod "a983cf0a-e87a-4615-a048-df14fb6c2f45" (UID: "a983cf0a-e87a-4615-a048-df14fb6c2f45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.985405 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985389 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-model-cache" pod "a983cf0a-e87a-4615-a048-df14fb6c2f45" (UID: "a983cf0a-e87a-4615-a048-df14fb6c2f45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.985581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985428 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a983cf0a-e87a-4615-a048-df14fb6c2f45-tls-certs\" (UniqueName: \"kubernetes.io/secret/a983cf0a-e87a-4615-a048-df14fb6c2f45-tls-certs\") pod \"a983cf0a-e87a-4615-a048-df14fb6c2f45\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " Nov 03 14:32:04.985581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985483 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-model-cache\") pod \"723ac416-02d4-48a5-a50b-21fb78a3fce5\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " Nov 03 14:32:04.985581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985534 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-tmp-dir\") pod \"723ac416-02d4-48a5-a50b-21fb78a3fce5\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " Nov 03 14:32:04.985581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985571 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/723ac416-02d4-48a5-a50b-21fb78a3fce5-tls-certs\" (UniqueName: \"kubernetes.io/secret/723ac416-02d4-48a5-a50b-21fb78a3fce5-tls-certs\") pod \"723ac416-02d4-48a5-a50b-21fb78a3fce5\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " Nov 03 14:32:04.986645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985614 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-tmp-dir\") pod \"a983cf0a-e87a-4615-a048-df14fb6c2f45\" (UID: \"a983cf0a-e87a-4615-a048-df14fb6c2f45\") " Nov 03 14:32:04.986645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985659 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723ac416-02d4-48a5-a50b-21fb78a3fce5-kube-api-access-zkkvt\" (UniqueName: \"kubernetes.io/projected/723ac416-02d4-48a5-a50b-21fb78a3fce5-kube-api-access-zkkvt\") pod \"723ac416-02d4-48a5-a50b-21fb78a3fce5\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " Nov 03 14:32:04.986645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985693 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-dshm\") pod \"723ac416-02d4-48a5-a50b-21fb78a3fce5\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " Nov 03 14:32:04.986645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.985726 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-home\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-home\") pod \"723ac416-02d4-48a5-a50b-21fb78a3fce5\" (UID: \"723ac416-02d4-48a5-a50b-21fb78a3fce5\") " Nov 03 14:32:04.986645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.986018 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:04.986645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.986038 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:04.986645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.986363 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-tmp-dir" pod "a983cf0a-e87a-4615-a048-df14fb6c2f45" (UID: "a983cf0a-e87a-4615-a048-df14fb6c2f45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.988171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.987312 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-model-cache" pod "723ac416-02d4-48a5-a50b-21fb78a3fce5" (UID: "723ac416-02d4-48a5-a50b-21fb78a3fce5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.988171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.987997 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-home" pod "723ac416-02d4-48a5-a50b-21fb78a3fce5" (UID: "723ac416-02d4-48a5-a50b-21fb78a3fce5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.988171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.988052 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-tmp-dir" pod "723ac416-02d4-48a5-a50b-21fb78a3fce5" (UID: "723ac416-02d4-48a5-a50b-21fb78a3fce5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.988171 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.988140 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-dshm" pod "a983cf0a-e87a-4615-a048-df14fb6c2f45" (UID: "a983cf0a-e87a-4615-a048-df14fb6c2f45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.988374 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.988196 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a983cf0a-e87a-4615-a048-df14fb6c2f45-tls-certs" pod "a983cf0a-e87a-4615-a048-df14fb6c2f45" (UID: "a983cf0a-e87a-4615-a048-df14fb6c2f45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:04.988374 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.988224 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a983cf0a-e87a-4615-a048-df14fb6c2f45-kube-api-access-9bjl4" pod "a983cf0a-e87a-4615-a048-df14fb6c2f45" (UID: "a983cf0a-e87a-4615-a048-df14fb6c2f45"). InnerVolumeSpecName "kube-api-access-9bjl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:04.989204 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.989181 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723ac416-02d4-48a5-a50b-21fb78a3fce5-kube-api-access-zkkvt" pod "723ac416-02d4-48a5-a50b-21fb78a3fce5" (UID: "723ac416-02d4-48a5-a50b-21fb78a3fce5"). InnerVolumeSpecName "kube-api-access-zkkvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:04.989469 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.989449 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/723ac416-02d4-48a5-a50b-21fb78a3fce5-tls-certs" pod "723ac416-02d4-48a5-a50b-21fb78a3fce5" (UID: "723ac416-02d4-48a5-a50b-21fb78a3fce5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:04.989937 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:04.989913 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-dshm" pod "723ac416-02d4-48a5-a50b-21fb78a3fce5" (UID: "723ac416-02d4-48a5-a50b-21fb78a3fce5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086879 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkkvt\" (UniqueName: \"kubernetes.io/projected/723ac416-02d4-48a5-a50b-21fb78a3fce5-kube-api-access-zkkvt\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086911 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086921 4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-home\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086930 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjl4\" (UniqueName: \"kubernetes.io/projected/a983cf0a-e87a-4615-a048-df14fb6c2f45-kube-api-access-9bjl4\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086939 4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-dshm\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086947 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a983cf0a-e87a-4615-a048-df14fb6c2f45-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086959 4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-model-cache\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086971 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723ac416-02d4-48a5-a50b-21fb78a3fce5-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.086969 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086979 4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723ac416-02d4-48a5-a50b-21fb78a3fce5-tls-certs\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.087933 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.086988 4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a983cf0a-e87a-4615-a048-df14fb6c2f45-tmp-dir\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.373039 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.372950 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2"] Nov 03 14:32:05.379211 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.379180 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-w2-cdb532-kserve-6565dcd496-pbcn2"] Nov 03 14:32:05.481486 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.481439 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc"] Nov 03 14:32:05.484767 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:05.484738 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/distinct-2-w2-cdb532-kserve-79fc47d587-fw6tc"] Nov 03 14:32:06.950096 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:06.950008 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723ac416-02d4-48a5-a50b-21fb78a3fce5" path="/var/lib/kubelet/pods/723ac416-02d4-48a5-a50b-21fb78a3fce5/volumes" Nov 03 14:32:06.950485 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:06.950462 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a983cf0a-e87a-4615-a048-df14fb6c2f45" path="/var/lib/kubelet/pods/a983cf0a-e87a-4615-a048-df14fb6c2f45/volumes" Nov 03 14:32:08.982357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:08.982323 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:10.015499 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:10.015454 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:15.567700 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:15.567655 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:15.697988 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:15.697951 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:15.924972 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:15.924930 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:18.558372 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:32:18.558333 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:32:26.814782 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:26.814748 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:27.218847 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:27.218810 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:27.349663 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:27.349629 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:27.563321 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:27.563243 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:30.100860 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:30.100817 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:30.594132 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:30.594101 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:32.109214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:32.109182 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qkwsd_1d3c8424-3ae1-488e-b37f-1d2a5d6c5199/manager/0.log" Nov 03 14:32:32.427025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:32.426993 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fvdmt_199f6b24-0d49-4637-b048-17b035f48627/manager/0.log" Nov 03 14:32:33.052121 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:33.052063 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8h2h7_27a5a362-b740-42a2-8f27-c5a918dfba78/manager/0.log" Nov 03 14:32:33.410832 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:33.410799 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:33.546059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:33.546019 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:33.710258 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:33.710175 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mvjp9_7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a/manager/0.log" Nov 03 14:32:35.089162 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:35.089130 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwnhg_19a01435-9815-42ed-9e2d-324a54f125bd/manager/0.log" Nov 03 14:32:35.321497 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:35.321467 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8kq74_dfd95a5f-d506-4461-8baf-44d26ec69181/manager/0.log" Nov 03 14:32:35.605413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:35.605363 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xbk2m_bde28c41-ee98-42d6-ad61-6d08466d2fc9/manager/0.log" Nov 03 14:32:36.553609 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:36.553575 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-p9scb_d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb/server/0.log" Nov 03 14:32:36.752661 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:36.752626 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:36.781213 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:36.781168 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pghpt_3ff5388d-d285-446d-9ccc-98a7be123067/manager/0.log" Nov 03 14:32:42.001393 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:42.001340 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:42.625828 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:42.625787 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:42.853937 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:42.853897 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:45.958611 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:45.958575 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:46.151430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:46.151384 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:46.298202 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:46.298115 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:48.557012 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:32:48.556970 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:32:49.295063 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:49.295023 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:49.470154 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:49.470115 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:52.629746 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:52.629709 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:52.775185 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:52.775150 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:52.917740 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:52.917701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:32:57.965259 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:32:57.965221 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:03.236192 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:03.236146 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:08.393841 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:08.393804 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:14.223372 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:14.223339 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:18.556986 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:33:18.556949 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:33:19.477353 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:19.477319 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:22.483203 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483167 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w"] Nov 03 14:33:22.483600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483551 4924 cpu_manager.go:414] "containerMap: removing container" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" Nov 03 14:33:22.483600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483564 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" Nov 03 14:33:22.483600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483573 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.483600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483578 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.483600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483589 4924 cpu_manager.go:414] "containerMap: removing container" podUID="9839ceb3-a0c7-4748-9f6b-307edd6bf097" containerName="maas-api" Nov 03 14:33:22.483600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483594 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9839ceb3-a0c7-4748-9f6b-307edd6bf097" containerName="maas-api" Nov 03 14:33:22.483600 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483602 4924 cpu_manager.go:414] "containerMap: removing container" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483608 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483619 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d47013a3-ed77-44a3-917b-b05d224cd6f5" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483624 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47013a3-ed77-44a3-917b-b05d224cd6f5" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483633 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483637 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483647 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a983cf0a-e87a-4615-a048-df14fb6c2f45" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483651 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a983cf0a-e87a-4615-a048-df14fb6c2f45" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483656 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483660 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483667 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483671 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483678 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483682 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483695 4924 cpu_manager.go:414] "containerMap: removing container" podUID="723ac416-02d4-48a5-a50b-21fb78a3fce5" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483699 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723ac416-02d4-48a5-a50b-21fb78a3fce5" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483708 4924 cpu_manager.go:414] "containerMap: removing container" podUID="077a2a59-e1a7-4f11-af5d-4373cc6643ad" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483714 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077a2a59-e1a7-4f11-af5d-4373cc6643ad" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483722 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a2ff7954-e696-4dcf-90f9-ee949cd35c27" containerName="maas-api" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483727 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ff7954-e696-4dcf-90f9-ee949cd35c27" containerName="maas-api" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483736 4924 cpu_manager.go:414] "containerMap: removing container" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483740 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483748 4924 cpu_manager.go:414] "containerMap: removing container" podUID="b26d3697-b507-4e1b-9add-ceca7dac937a" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483753 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26d3697-b507-4e1b-9add-ceca7dac937a" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483807 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="723ac416-02d4-48a5-a50b-21fb78a3fce5" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483816 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="077a2a59-e1a7-4f11-af5d-4373cc6643ad" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483822 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ff7954-e696-4dcf-90f9-ee949cd35c27" containerName="maas-api" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483828 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ea1109-ec3c-4f0f-9d5e-1566441f2b1e" containerName="istio-proxy" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483833 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26d3697-b507-4e1b-9add-ceca7dac937a" containerName="main" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483839 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb7e333-ae06-4712-acf2-792344bbd8eb" containerName="istio-proxy" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483845 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.483838 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483851 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9839ceb3-a0c7-4748-9f6b-307edd6bf097" containerName="maas-api" Nov 03 14:33:22.484803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483857 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b82e11-cb81-4df8-8d5e-65398626aaae" containerName="main" Nov 03 14:33:22.484803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483862 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47013a3-ed77-44a3-917b-b05d224cd6f5" containerName="main" Nov 03 14:33:22.484803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483868 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.484803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483874 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a983cf0a-e87a-4615-a048-df14fb6c2f45" containerName="main" Nov 03 14:33:22.484803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483879 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" Nov 03 14:33:22.484803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483883 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa8e6e7-629c-4479-86ab-e810221ef398" containerName="istio-proxy" Nov 03 14:33:22.484803 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.483888 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c95f3a8-91d3-4a5c-8f8e-d4103a9c2a4b" containerName="cleanup" Nov 03 14:33:22.487124 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.487108 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.540730 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.540686 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w"] Nov 03 14:33:22.569319 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.569285 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.569512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.569330 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.569512 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.569466 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmttz\" (UniqueName: \"kubernetes.io/projected/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-kube-api-access-tmttz\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.584360 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.584331 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:33:22.584482 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.584428 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:33:22.670756 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.670722 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmttz\" (UniqueName: \"kubernetes.io/projected/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-kube-api-access-tmttz\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.670941 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.670835 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.670941 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.670881 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.672112 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.672055 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.675975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.674165 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.724924 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.724896 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmttz\" (UniqueName: \"kubernetes.io/projected/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-kube-api-access-tmttz\") pod \"maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:22.800047 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:22.799970 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:23.163964 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:23.163940 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w"] Nov 03 14:33:23.166501 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:33:23.166474 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf184d8a6_131b_4b3e_bec4_7bc955d3dc5f.slice/crio-f260a8368f30206a315ac524272531aa2353f9f4ec43b1ee946e78f52a83249d WatchSource:0}: Error finding container f260a8368f30206a315ac524272531aa2353f9f4ec43b1ee946e78f52a83249d: Status 404 returned error can't find the container with id f260a8368f30206a315ac524272531aa2353f9f4ec43b1ee946e78f52a83249d Nov 03 14:33:23.385436 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:23.385400 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" event={"ID":"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f","Type":"ContainerStarted","Data":"f260a8368f30206a315ac524272531aa2353f9f4ec43b1ee946e78f52a83249d"} Nov 03 14:33:24.392224 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:24.392188 4924 generic.go:356] "Generic (PLEG): container finished" podID="f184d8a6-131b-4b3e-bec4-7bc955d3dc5f" containerID="8e9218cde5987f6cb7156b9a98ad43c3a5f6a802c81908cafa8e0e6e0f9dbead" exitCode=0 Nov 03 14:33:24.392656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:24.392264 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" event={"ID":"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f","Type":"ContainerDied","Data":"8e9218cde5987f6cb7156b9a98ad43c3a5f6a802c81908cafa8e0e6e0f9dbead"} Nov 03 14:33:24.670587 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:24.670546 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:25.530975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.530946 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:25.594332 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.594300 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-openshift-ca\") pod \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " Nov 03 14:33:25.594503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.594350 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-xks-ca\") pod \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " Nov 03 14:33:25.594503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.594377 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-kube-api-access-tmttz\" (UniqueName: \"kubernetes.io/projected/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-kube-api-access-tmttz\") pod \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\" (UID: \"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f\") " Nov 03 14:33:25.594714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.594692 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-openshift-ca" pod "f184d8a6-131b-4b3e-bec4-7bc955d3dc5f" (UID: "f184d8a6-131b-4b3e-bec4-7bc955d3dc5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:25.596523 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.596491 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-xks-ca" pod "f184d8a6-131b-4b3e-bec4-7bc955d3dc5f" (UID: "f184d8a6-131b-4b3e-bec4-7bc955d3dc5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:25.596523 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.596507 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-kube-api-access-tmttz" pod "f184d8a6-131b-4b3e-bec4-7bc955d3dc5f" (UID: "f184d8a6-131b-4b3e-bec4-7bc955d3dc5f"). InnerVolumeSpecName "kube-api-access-tmttz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:25.695214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.695182 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:25.695214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.695211 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:25.695214 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:25.695221 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmttz\" (UniqueName: \"kubernetes.io/projected/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f-kube-api-access-tmttz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:26.404573 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:26.404545 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" Nov 03 14:33:26.404791 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:26.404536 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w" event={"ID":"f184d8a6-131b-4b3e-bec4-7bc955d3dc5f","Type":"ContainerDied","Data":"f260a8368f30206a315ac524272531aa2353f9f4ec43b1ee946e78f52a83249d"} Nov 03 14:33:26.404791 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:26.404683 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f260a8368f30206a315ac524272531aa2353f9f4ec43b1ee946e78f52a83249d" Nov 03 14:33:29.765387 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:29.765349 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:34.272826 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:34.272787 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:34.426503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:34.426463 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:34.554501 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:34.554411 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:38.590764 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:38.590681 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:38.855754 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:38.855668 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:39.056052 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:39.056016 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w"] Nov 03 14:33:39.102730 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:39.102694 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f530cd7c-6p97w"] Nov 03 14:33:40.949174 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:40.949138 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f184d8a6-131b-4b3e-bec4-7bc955d3dc5f" path="/var/lib/kubelet/pods/f184d8a6-131b-4b3e-bec4-7bc955d3dc5f/volumes" Nov 03 14:33:43.603518 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:43.603487 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:46.518779 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:33:46.518741 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:33:48.557967 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:33:48.557923 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:34:18.558109 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:34:18.558053 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:34:23.343141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:23.343094 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:34:25.772192 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:25.772148 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:34:27.844488 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:27.844456 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:34:27.866778 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:27.866752 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:34:30.768853 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.768807 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482"] Nov 03 14:34:30.769278 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.769230 4924 cpu_manager.go:414] "containerMap: removing container" podUID="f184d8a6-131b-4b3e-bec4-7bc955d3dc5f" containerName="revoke-keys" Nov 03 14:34:30.769278 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.769244 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f184d8a6-131b-4b3e-bec4-7bc955d3dc5f" containerName="revoke-keys" Nov 03 14:34:30.769354 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.769331 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f184d8a6-131b-4b3e-bec4-7bc955d3dc5f" containerName="revoke-keys" Nov 03 14:34:30.772599 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.772576 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.821856 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.821829 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wd2hr\"" Nov 03 14:34:30.870918 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.870883 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482"] Nov 03 14:34:30.876481 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876453 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f586dcb0-04d2-4152-9556-931824aab29b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.876654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876486 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.876654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876504 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f586dcb0-04d2-4152-9556-931824aab29b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.876654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876520 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f586dcb0-04d2-4152-9556-931824aab29b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.876654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876586 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.876654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876624 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.876954 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876685 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.876954 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876724 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.876954 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.876800 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8pb4\" (UniqueName: \"kubernetes.io/projected/f586dcb0-04d2-4152-9556-931824aab29b-kube-api-access-m8pb4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.977911 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.977873 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.977934 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8pb4\" (UniqueName: \"kubernetes.io/projected/f586dcb0-04d2-4152-9556-931824aab29b-kube-api-access-m8pb4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.977964 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f586dcb0-04d2-4152-9556-931824aab29b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.977982 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.977999 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f586dcb0-04d2-4152-9556-931824aab29b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978131 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f586dcb0-04d2-4152-9556-931824aab29b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978166 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978191 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978337 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978228 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978339 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978641 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978581 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978641 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978630 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978772 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978668 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.978879 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.978856 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f586dcb0-04d2-4152-9556-931824aab29b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.980338 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.980318 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f586dcb0-04d2-4152-9556-931824aab29b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:30.980571 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:30.980553 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f586dcb0-04d2-4152-9556-931824aab29b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:31.032900 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.032821 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f586dcb0-04d2-4152-9556-931824aab29b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:31.033039 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.032913 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8pb4\" (UniqueName: \"kubernetes.io/projected/f586dcb0-04d2-4152-9556-931824aab29b-kube-api-access-m8pb4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-sw482\" (UID: \"f586dcb0-04d2-4152-9556-931824aab29b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:31.086207 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.086176 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:31.504421 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.504396 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482"] Nov 03 14:34:31.506486 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:34:31.506450 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586dcb0_04d2_4152_9556_931824aab29b.slice/crio-20309ec01a1ea91dd960abff4d827207f0de5be87eb760c761ddddd8cc8ac019 WatchSource:0}: Error finding container 20309ec01a1ea91dd960abff4d827207f0de5be87eb760c761ddddd8cc8ac019: Status 404 returned error can't find the container with id 20309ec01a1ea91dd960abff4d827207f0de5be87eb760c761ddddd8cc8ac019 Nov 03 14:34:31.508524 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.508490 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:34:31.508612 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.508554 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:34:31.508612 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.508585 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:34:31.752641 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.752592 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" event={"ID":"f586dcb0-04d2-4152-9556-931824aab29b","Type":"ContainerStarted","Data":"75181d559a6cc523eb066268e56ffaa2fee91f716eca1aedd1a27c333001561e"} Nov 03 14:34:31.752858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.752690 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" event={"ID":"f586dcb0-04d2-4152-9556-931824aab29b","Type":"ContainerStarted","Data":"20309ec01a1ea91dd960abff4d827207f0de5be87eb760c761ddddd8cc8ac019"} Nov 03 14:34:31.864831 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:31.864716 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podStartSLOduration=1.8646991320000001 podStartE2EDuration="1.864699132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:31.863421737 +0000 UTC m=+5405.057825331" watchObservedRunningTime="2026-11-03 14:34:31.864699132 +0000 UTC m=+5405.059102728" Nov 03 14:34:32.086450 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:32.086412 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:32.088044 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:32.088017 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 14:34:32.088182 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:32.088114 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 14:34:33.086868 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:33.086834 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 14:34:33.087288 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:33.086897 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 14:34:34.086476 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:34.086431 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 14:34:34.086655 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:34.086508 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 14:34:35.086908 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:35.086873 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 14:34:35.087326 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:35.086940 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 14:34:36.086708 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:36.086672 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 14:34:36.086937 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:36.086733 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 14:34:37.087212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:37.087177 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 14:34:37.087645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:37.087231 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 14:34:38.086615 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:38.086573 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 14:34:38.086795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:38.086656 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 14:34:39.852788 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:39.852741 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:39.853324 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:39.853112 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:39.853974 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:39.853952 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" Nov 03 14:34:46.845009 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:34:46.844969 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:34:48.557879 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:34:48.557842 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:35:12.384814 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:12.384719 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:35:12.646206 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:12.646121 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:35:16.151917 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.151868 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv"] Nov 03 14:35:16.156388 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.156364 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.196453 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.196419 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt"] Nov 03 14:35:16.200000 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.199974 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.249672 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.249626 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv"] Nov 03 14:35:16.297738 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.297698 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt"] Nov 03 14:35:16.310940 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.310911 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.311141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.310943 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.310962 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43079ebb-3c46-4416-8822-bbe2be66119b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.310983 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfpp\" (UniqueName: \"kubernetes.io/projected/43079ebb-3c46-4416-8822-bbe2be66119b-kube-api-access-7cfpp\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311141 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311055 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311368 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311163 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cb7330-19b8-4112-b96b-cc70285298ed-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.311368 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311197 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.311368 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311213 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cb7330-19b8-4112-b96b-cc70285298ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.311368 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311230 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311368 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311353 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43079ebb-3c46-4416-8822-bbe2be66119b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311403 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311457 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311488 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.311666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311550 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.311666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311578 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43079ebb-3c46-4416-8822-bbe2be66119b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.311666 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311646 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cb7330-19b8-4112-b96b-cc70285298ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.311883 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311716 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q555m\" (UniqueName: \"kubernetes.io/projected/05cb7330-19b8-4112-b96b-cc70285298ed-kube-api-access-q555m\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.311883 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.311743 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.412514 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412476 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.412706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412534 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.412706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412581 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.412706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412604 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43079ebb-3c46-4416-8822-bbe2be66119b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.412706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412632 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfpp\" (UniqueName: \"kubernetes.io/projected/43079ebb-3c46-4416-8822-bbe2be66119b-kube-api-access-7cfpp\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.412706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412662 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.412706 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412688 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cb7330-19b8-4112-b96b-cc70285298ed-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412729 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412755 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cb7330-19b8-4112-b96b-cc70285298ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412780 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412825 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43079ebb-3c46-4416-8822-bbe2be66119b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412860 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412897 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412927 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412939 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.412955 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413000 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43079ebb-3c46-4416-8822-bbe2be66119b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.413093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413059 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cb7330-19b8-4112-b96b-cc70285298ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413128 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q555m\" (UniqueName: \"kubernetes.io/projected/05cb7330-19b8-4112-b96b-cc70285298ed-kube-api-access-q555m\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413224 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.413721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413301 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.413721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413303 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.413721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413509 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43079ebb-3c46-4416-8822-bbe2be66119b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.413721 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413570 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.414101 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.413828 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.414101 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.414026 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cb7330-19b8-4112-b96b-cc70285298ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.414101 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.414051 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.414239 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.414215 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.415596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.415570 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cb7330-19b8-4112-b96b-cc70285298ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.415596 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.415588 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43079ebb-3c46-4416-8822-bbe2be66119b-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.415737 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.415681 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cb7330-19b8-4112-b96b-cc70285298ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.415779 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.415734 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43079ebb-3c46-4416-8822-bbe2be66119b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.467405 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.467378 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43079ebb-3c46-4416-8822-bbe2be66119b-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.467405 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.467379 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q555m\" (UniqueName: \"kubernetes.io/projected/05cb7330-19b8-4112-b96b-cc70285298ed-kube-api-access-q555m\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.467593 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.467388 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfpp\" (UniqueName: \"kubernetes.io/projected/43079ebb-3c46-4416-8822-bbe2be66119b-kube-api-access-7cfpp\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dx6qt\" (UID: \"43079ebb-3c46-4416-8822-bbe2be66119b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.467593 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.467487 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cb7330-19b8-4112-b96b-cc70285298ed-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nmndv\" (UID: \"05cb7330-19b8-4112-b96b-cc70285298ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.473361 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.473338 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:16.512783 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.512750 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:16.889993 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.889963 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv"] Nov 03 14:35:16.891554 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:35:16.891522 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cb7330_19b8_4112_b96b_cc70285298ed.slice/crio-689ef0a16395246357e560375cbb57fd33c30a4f83e0d6a1abcd1481c6773b08 WatchSource:0}: Error finding container 689ef0a16395246357e560375cbb57fd33c30a4f83e0d6a1abcd1481c6773b08: Status 404 returned error can't find the container with id 689ef0a16395246357e560375cbb57fd33c30a4f83e0d6a1abcd1481c6773b08 Nov 03 14:35:16.893573 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.893541 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:16.893651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.893606 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:16.893651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.893633 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:16.940221 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.940019 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt"] Nov 03 14:35:16.941142 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:35:16.941115 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43079ebb_3c46_4416_8822_bbe2be66119b.slice/crio-310ab46df683367b2f77e65e0dfe394ee26539ec9368f580411cfae0891bdcf0 WatchSource:0}: Error finding container 310ab46df683367b2f77e65e0dfe394ee26539ec9368f580411cfae0891bdcf0: Status 404 returned error can't find the container with id 310ab46df683367b2f77e65e0dfe394ee26539ec9368f580411cfae0891bdcf0 Nov 03 14:35:16.943547 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.943508 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:16.943640 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.943595 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:16.943705 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.943640 4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:16.997836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.997736 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" event={"ID":"05cb7330-19b8-4112-b96b-cc70285298ed","Type":"ContainerStarted","Data":"6ba603e84c93b695739f3f6855d5ec3246a46d4085f1717ed614c75faa1581ab"} Nov 03 14:35:16.997836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:16.997803 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" event={"ID":"05cb7330-19b8-4112-b96b-cc70285298ed","Type":"ContainerStarted","Data":"689ef0a16395246357e560375cbb57fd33c30a4f83e0d6a1abcd1481c6773b08"} Nov 03 14:35:17.000237 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:17.000154 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" event={"ID":"43079ebb-3c46-4416-8822-bbe2be66119b","Type":"ContainerStarted","Data":"310ab46df683367b2f77e65e0dfe394ee26539ec9368f580411cfae0891bdcf0"} Nov 03 14:35:17.116090 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:17.115912 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podStartSLOduration=1.11589119 podStartE2EDuration="1.11589119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:17.112543553 +0000 UTC m=+5450.306947232" watchObservedRunningTime="2026-11-03 14:35:17.11589119 +0000 UTC m=+5450.310294789" Nov 03 14:35:17.476187 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:17.473824 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:17.477476 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:17.477442 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 14:35:17.477670 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:17.477494 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 14:35:18.006400 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:18.006361 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" event={"ID":"43079ebb-3c46-4416-8822-bbe2be66119b","Type":"ContainerStarted","Data":"31259c0e88777e1375e2fbf7b128fc2f28d9da2a0e2405987567a3e47469085e"} Nov 03 14:35:18.124370 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:18.124315 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podStartSLOduration=2.1242964 podStartE2EDuration="2.1242964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:18.123846592 +0000 UTC m=+5451.318250185" watchObservedRunningTime="2026-11-03 14:35:18.1242964 +0000 UTC m=+5451.318699993" Nov 03 14:35:18.477093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:18.476487 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 14:35:18.477093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:18.476555 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 14:35:18.514038 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:18.514004 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:18.515411 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:18.515386 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 14:35:18.515587 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:18.515562 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 14:35:18.596170 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:35:18.596130 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:35:19.476111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:19.475519 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 14:35:19.476111 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:19.475587 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 14:35:19.514014 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:19.513975 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 14:35:19.514437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:19.514036 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 14:35:20.475579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:20.475533 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 14:35:20.475882 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:20.475851 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 14:35:20.514276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:20.514235 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 14:35:20.514664 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:20.514285 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 14:35:21.476247 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:21.476207 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 14:35:21.476479 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:21.476277 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 14:35:21.513597 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:21.513556 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 14:35:21.513777 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:21.513613 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 14:35:22.475659 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:22.475602 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 14:35:22.476246 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:22.476182 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 14:35:22.513404 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:22.513371 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 14:35:22.513584 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:22.513427 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 14:35:23.473686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:23.473651 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 14:35:23.473915 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:23.473708 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 14:35:23.513510 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:23.513470 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 14:35:23.513907 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:23.513539 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 14:35:24.474659 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:24.474614 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 14:35:24.474858 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:24.474701 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 14:35:24.514130 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:24.514088 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 14:35:24.514528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:24.514154 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 14:35:26.477096 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:26.476496 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:26.477096 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:26.476559 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:26.554725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:26.554295 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:26.554725 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:26.554365 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:27.521227 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:27.521184 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:27.521731 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:27.521247 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:27.606272 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:27.606229 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:27.606480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:27.606293 4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:28.352001 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:28.351967 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:28.353146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:28.352990 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:28.353801 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:28.353783 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" Nov 03 14:35:28.500553 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:28.500520 4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:28.500864 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:28.500841 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:28.549950 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:28.549914 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" Nov 03 14:35:47.234588 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:47.234546 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:35:47.367884 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:35:47.367848 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:35:48.558425 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:35:48.558390 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:36:16.597459 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:36:16.597423 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:36:17.163645 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:36:17.163595 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:36:18.559797 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:36:18.559752 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:36:20.525289 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:36:20.525252 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:36:32.517489 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:36:32.517455 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:36:32.644135 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:36:32.644100 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:36:33.305756 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:36:33.305725 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:36:33.427866 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:36:33.427818 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:36:47.563482 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:36:47.563397 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:36:48.559001 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:36:48.558965 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:37:13.306161 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.306124 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z"] Nov 03 14:37:13.309821 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.309802 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.361210 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.361166 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:37:13.364546 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.364519 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z"] Nov 03 14:37:13.389789 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.389759 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.389936 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.389816 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.389936 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.389849 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzl4\" (UniqueName: \"kubernetes.io/projected/044696bf-7f34-4e34-b281-c09974e28af5-kube-api-access-jjzl4\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.407494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.407467 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:37:13.490715 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.490678 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.490891 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.490735 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzl4\" (UniqueName: \"kubernetes.io/projected/044696bf-7f34-4e34-b281-c09974e28af5-kube-api-access-jjzl4\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.490891 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.490783 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.491513 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.491481 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.493135 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.493112 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.544325 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.544298 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzl4\" (UniqueName: \"kubernetes.io/projected/044696bf-7f34-4e34-b281-c09974e28af5-kube-api-access-jjzl4\") pod \"maas-api-revoke-sub-cfbc0f89-nrr9z\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.622523 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.622444 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:13.991465 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:13.991438 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z"] Nov 03 14:37:13.993644 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:37:13.993613 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044696bf_7f34_4e34_b281_c09974e28af5.slice/crio-8bacdc29c61e6f2e661033882720940fbf2c5abc9799b9207d0ab2e1acf01450 WatchSource:0}: Error finding container 8bacdc29c61e6f2e661033882720940fbf2c5abc9799b9207d0ab2e1acf01450: Status 404 returned error can't find the container with id 8bacdc29c61e6f2e661033882720940fbf2c5abc9799b9207d0ab2e1acf01450 Nov 03 14:37:14.768965 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:14.768923 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" event={"ID":"044696bf-7f34-4e34-b281-c09974e28af5","Type":"ContainerStarted","Data":"0a819f4c51e743f9e09a056e5f7f086dc02b422edb76944b903b5ac202c7f891"} Nov 03 14:37:14.769445 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:14.768980 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" event={"ID":"044696bf-7f34-4e34-b281-c09974e28af5","Type":"ContainerStarted","Data":"8bacdc29c61e6f2e661033882720940fbf2c5abc9799b9207d0ab2e1acf01450"} Nov 03 14:37:14.880128 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:14.880061 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" podStartSLOduration=1.880047051 podStartE2EDuration="1.880047051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:14.877779795 +0000 UTC m=+5568.072183389" watchObservedRunningTime="2026-11-03 14:37:14.880047051 +0000 UTC m=+5568.074450670" Nov 03 14:37:15.775543 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:15.775500 4924 generic.go:356] "Generic (PLEG): container finished" podID="044696bf-7f34-4e34-b281-c09974e28af5" containerID="0a819f4c51e743f9e09a056e5f7f086dc02b422edb76944b903b5ac202c7f891" exitCode=0 Nov 03 14:37:15.775938 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:15.775573 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" event={"ID":"044696bf-7f34-4e34-b281-c09974e28af5","Type":"ContainerDied","Data":"0a819f4c51e743f9e09a056e5f7f086dc02b422edb76944b903b5ac202c7f891"} Nov 03 14:37:16.923039 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:16.923015 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:17.022782 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.022744 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-openshift-ca\") pod \"044696bf-7f34-4e34-b281-c09974e28af5\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " Nov 03 14:37:17.022967 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.022813 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044696bf-7f34-4e34-b281-c09974e28af5-kube-api-access-jjzl4\" (UniqueName: \"kubernetes.io/projected/044696bf-7f34-4e34-b281-c09974e28af5-kube-api-access-jjzl4\") pod \"044696bf-7f34-4e34-b281-c09974e28af5\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " Nov 03 14:37:17.022967 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.022860 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-xks-ca\") pod \"044696bf-7f34-4e34-b281-c09974e28af5\" (UID: \"044696bf-7f34-4e34-b281-c09974e28af5\") " Nov 03 14:37:17.023168 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.023145 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-openshift-ca" pod "044696bf-7f34-4e34-b281-c09974e28af5" (UID: "044696bf-7f34-4e34-b281-c09974e28af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:37:17.023276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.023251 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:17.024975 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.024947 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-xks-ca" pod "044696bf-7f34-4e34-b281-c09974e28af5" (UID: "044696bf-7f34-4e34-b281-c09974e28af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:37:17.025059 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.025012 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044696bf-7f34-4e34-b281-c09974e28af5-kube-api-access-jjzl4" pod "044696bf-7f34-4e34-b281-c09974e28af5" (UID: "044696bf-7f34-4e34-b281-c09974e28af5"). InnerVolumeSpecName "kube-api-access-jjzl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:17.124142 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.124043 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjzl4\" (UniqueName: \"kubernetes.io/projected/044696bf-7f34-4e34-b281-c09974e28af5-kube-api-access-jjzl4\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:17.124142 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.124099 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044696bf-7f34-4e34-b281-c09974e28af5-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:17.794285 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.794251 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" Nov 03 14:37:17.794483 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.794279 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z" event={"ID":"044696bf-7f34-4e34-b281-c09974e28af5","Type":"ContainerDied","Data":"8bacdc29c61e6f2e661033882720940fbf2c5abc9799b9207d0ab2e1acf01450"} Nov 03 14:37:17.794483 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:17.794361 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bacdc29c61e6f2e661033882720940fbf2c5abc9799b9207d0ab2e1acf01450" Nov 03 14:37:18.557821 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:37:18.557788 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:37:26.448254 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:26.448215 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:26.594109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:26.594053 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:28.715134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:28.715092 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:28.830241 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:28.830203 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:31.982799 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:31.982762 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc"] Nov 03 14:37:31.983270 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:31.983255 4924 cpu_manager.go:414] "containerMap: removing container" podUID="044696bf-7f34-4e34-b281-c09974e28af5" containerName="revoke-keys" Nov 03 14:37:31.983318 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:31.983271 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044696bf-7f34-4e34-b281-c09974e28af5" containerName="revoke-keys" Nov 03 14:37:31.983386 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:31.983376 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="044696bf-7f34-4e34-b281-c09974e28af5" containerName="revoke-keys" Nov 03 14:37:31.990821 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:31.990795 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.042276 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.042241 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc"] Nov 03 14:37:32.067579 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.067537 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.067782 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.067658 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgsp\" (UniqueName: \"kubernetes.io/projected/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-kube-api-access-9mgsp\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.067852 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.067800 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.085659 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.085629 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:37:32.085809 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.085760 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:37:32.169042 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.169000 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.169237 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.169052 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgsp\" (UniqueName: \"kubernetes.io/projected/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-kube-api-access-9mgsp\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.169237 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.169128 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.169750 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.169719 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.171560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.171536 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.224282 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.224246 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgsp\" (UniqueName: \"kubernetes.io/projected/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-kube-api-access-9mgsp\") pod \"maas-api-revoke-sub-f8b5238d-p7tvc\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.304131 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.304023 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:32.675481 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.675453 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc"] Nov 03 14:37:32.677106 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:37:32.677050 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96c72f6_9a2d_4d91_95c5_c603ed3cda18.slice/crio-ef26409c812046e196570c0e26bbf8e69831ba5c82dc875a853fbb61da50332c WatchSource:0}: Error finding container ef26409c812046e196570c0e26bbf8e69831ba5c82dc875a853fbb61da50332c: Status 404 returned error can't find the container with id ef26409c812046e196570c0e26bbf8e69831ba5c82dc875a853fbb61da50332c Nov 03 14:37:32.875829 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:32.875791 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" event={"ID":"a96c72f6-9a2d-4d91-95c5-c603ed3cda18","Type":"ContainerStarted","Data":"ef26409c812046e196570c0e26bbf8e69831ba5c82dc875a853fbb61da50332c"} Nov 03 14:37:33.901709 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:33.901672 4924 generic.go:356] "Generic (PLEG): container finished" podID="a96c72f6-9a2d-4d91-95c5-c603ed3cda18" containerID="aaf83c8db42124a1912de7c96fa7e37021d0f29460ad2ccce6fde2524431caa4" exitCode=0 Nov 03 14:37:33.902390 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:33.902361 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" event={"ID":"a96c72f6-9a2d-4d91-95c5-c603ed3cda18","Type":"ContainerDied","Data":"aaf83c8db42124a1912de7c96fa7e37021d0f29460ad2ccce6fde2524431caa4"} Nov 03 14:37:35.318260 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.318226 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:35.420999 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.420954 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-xks-ca\") pod \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " Nov 03 14:37:35.421225 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.421089 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-kube-api-access-9mgsp\" (UniqueName: \"kubernetes.io/projected/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-kube-api-access-9mgsp\") pod \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " Nov 03 14:37:35.421225 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.421184 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-openshift-ca\") pod \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\" (UID: \"a96c72f6-9a2d-4d91-95c5-c603ed3cda18\") " Nov 03 14:37:35.421743 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.421710 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-openshift-ca" pod "a96c72f6-9a2d-4d91-95c5-c603ed3cda18" (UID: "a96c72f6-9a2d-4d91-95c5-c603ed3cda18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:37:35.423342 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.423314 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-xks-ca" pod "a96c72f6-9a2d-4d91-95c5-c603ed3cda18" (UID: "a96c72f6-9a2d-4d91-95c5-c603ed3cda18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:37:35.423533 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.423511 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-kube-api-access-9mgsp" pod "a96c72f6-9a2d-4d91-95c5-c603ed3cda18" (UID: "a96c72f6-9a2d-4d91-95c5-c603ed3cda18"). InnerVolumeSpecName "kube-api-access-9mgsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:35.522372 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.522232 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mgsp\" (UniqueName: \"kubernetes.io/projected/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-kube-api-access-9mgsp\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:35.522372 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.522274 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:35.522372 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.522291 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96c72f6-9a2d-4d91-95c5-c603ed3cda18-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:35.916145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.916114 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" Nov 03 14:37:35.916145 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.916125 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc" event={"ID":"a96c72f6-9a2d-4d91-95c5-c603ed3cda18","Type":"ContainerDied","Data":"ef26409c812046e196570c0e26bbf8e69831ba5c82dc875a853fbb61da50332c"} Nov 03 14:37:35.916361 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:35.916179 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef26409c812046e196570c0e26bbf8e69831ba5c82dc875a853fbb61da50332c" Nov 03 14:37:44.652496 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:44.652458 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:45.300165 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:45.300123 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:45.516230 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:45.516193 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:47.862556 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:47.862520 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:48.559705 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:37:48.559668 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:37:49.439093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:49.437904 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:49.668135 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:49.668098 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:49.806896 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:49.806804 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:52.769872 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:52.769835 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:52.953681 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:52.953645 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:56.599315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:56.599280 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:56.774506 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:56.774472 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:37:56.896400 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:37:56.896313 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:01.955458 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:01.955421 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:07.161137 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:07.161022 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:12.503263 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:12.503223 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:14.253492 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:14.253447 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:14.897104 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:14.897048 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:15.028607 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:15.028573 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:15.181245 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:15.181206 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:17.974464 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:17.974422 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:18.468990 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:18.468944 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:18.562786 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:38:18.562750 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:38:23.170828 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:23.170792 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:28.373618 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:28.373576 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:33.456163 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:33.456128 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:33.581350 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:33.581318 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:37.385101 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:37.385051 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:38.672947 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:38.672907 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:43.769865 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:43.769828 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:48.183450 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:48.183399 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:48.307120 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:48.307058 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:48.557199 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:38:48.557119 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:38:48.972736 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:48.972701 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:54.184879 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:54.184834 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:38:59.379797 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:38:59.379761 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:04.465085 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:04.465037 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:09.670513 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:09.670480 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:14.402949 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:14.402875 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:14.776292 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:14.776261 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:14.914047 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:14.914015 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:18.557944 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:39:18.557899 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:39:19.456122 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:19.456089 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:19.965084 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:19.965037 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:20.939092 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:20.939037 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:22.034492 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:22.034450 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:22.175550 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:22.175518 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:25.355662 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:25.355624 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:26.911233 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:26.911200 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:27.882726 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:27.882699 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:39:27.906900 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:27.906874 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:39:28.867342 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:28.867314 4924 scope.go:122] "RemoveContainer" containerID="8e9218cde5987f6cb7156b9a98ad43c3a5f6a802c81908cafa8e0e6e0f9dbead" Nov 03 14:39:29.901120 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:29.901085 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:32.348698 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:32.348661 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:32.587743 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:32.587705 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:32.716353 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:32.716318 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:48.100468 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:48.100358 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:48.558180 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:39:48.558145 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:39:49.168577 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:49.168543 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:51.610343 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:51.610306 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:52.325316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:52.325283 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:55.220190 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:55.220154 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:39:55.356695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:39:55.356646 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:40:08.402988 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:40:08.402952 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:40:17.502186 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:40:17.502146 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:40:18.558264 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:40:18.558226 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:40:36.599916 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:40:36.599882 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:40:48.558106 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:40:48.558047 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:40:48.879939 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:40:48.879871 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:41:18.557557 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:41:18.557477 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:41:24.345355 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:41:24.345317 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:41:38.180351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:41:38.180316 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:41:48.557559 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:41:48.557518 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:41:49.213622 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:41:49.213590 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:41:49.328206 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:41:49.328170 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:42:17.048939 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:42:17.048899 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z"] Nov 03 14:42:17.095857 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:42:17.095824 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbc0f89-nrr9z"] Nov 03 14:42:18.558191 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:42:18.558146 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:42:18.949716 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:42:18.949680 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044696bf-7f34-4e34-b281-c09974e28af5" path="/var/lib/kubelet/pods/044696bf-7f34-4e34-b281-c09974e28af5/volumes" Nov 03 14:42:36.044045 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:42:36.044006 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc"] Nov 03 14:42:36.090620 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:42:36.090580 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b5238d-p7tvc"] Nov 03 14:42:36.949422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:42:36.949352 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96c72f6-9a2d-4d91-95c5-c603ed3cda18" path="/var/lib/kubelet/pods/a96c72f6-9a2d-4d91-95c5-c603ed3cda18/volumes" Nov 03 14:42:48.561149 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:42:48.561104 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:42:49.537695 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:42:49.537653 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:43:08.907417 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:43:08.907372 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:43:09.360028 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:43:09.359956 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:43:12.158125 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:43:12.158088 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:43:18.558123 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:43:18.558087 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:43:28.918178 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:43:28.918141 4924 scope.go:122] "RemoveContainer" containerID="0a819f4c51e743f9e09a056e5f7f086dc02b422edb76944b903b5ac202c7f891" Nov 03 14:43:48.557561 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:43:48.557519 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:43:49.883298 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:43:49.883257 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:44:18.558105 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:44:18.558012 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:44:27.920665 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:44:27.920622 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:44:27.945028 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:44:27.944999 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:44:28.940220 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:44:28.940180 4924 scope.go:122] "RemoveContainer" containerID="aaf83c8db42124a1912de7c96fa7e37021d0f29460ad2ccce6fde2524431caa4" Nov 03 14:44:48.558527 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:44:48.558487 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:44:50.208312 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:44:50.208274 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:44:51.201316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:44:51.201263 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:44:59.844959 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:44:59.844919 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:44:59.960905 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:44:59.960866 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:45:00.384465 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:45:00.384432 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:45:18.559099 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:45:18.559047 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:45:44.397357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:45:44.397244 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:45:44.398025 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:45:44.397972 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:45:44.547095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:45:44.546219 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:45:44.547095 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:45:44.546334 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:45:48.557130 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:45:48.557085 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:45:50.590656 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:45:50.590619 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:46:18.558239 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:46:18.558200 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:46:48.558551 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:46:48.558512 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:46:50.871856 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:46:50.871800 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:47:18.557748 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:47:18.557664 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:47:48.559021 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:47:48.558975 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:47:51.208456 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:47:51.208425 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:47:51.330383 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:47:51.330350 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:47:52.067429 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:47:52.067396 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:47:54.670904 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:47:54.670872 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:48:00.262218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:48:00.262181 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:48:18.558221 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:48:18.558174 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:48:48.633524 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:48:48.633292 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:48:51.551619 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:48:51.551570 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:48:55.906797 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:48:55.906749 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:48:55.907569 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:48:55.907534 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:49:18.557248 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:49:18.557205 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:49:27.958525 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:49:27.958485 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:49:27.984495 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:49:27.984465 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:49:37.751662 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:49:37.751609 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:49:48.557409 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:49:48.557372 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:49:52.004136 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:49:52.004094 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:18.558262 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:50:18.558174 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:50:23.081100 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:23.079946 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:23.866891 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:23.866860 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:24.174880 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:24.174844 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:28.433807 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:28.433766 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:28.646503 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:28.646451 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:28.813714 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:28.813630 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:31.794691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:31.794625 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:31.976462 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:31.976431 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:34.992810 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:34.992770 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:35.178442 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:35.178401 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:35.308620 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:35.308534 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:40.249576 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:40.249544 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:45.361581 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:45.361542 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:48.557952 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:50:48.557912 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:50:50.373451 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:50.373412 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:52.296591 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:52.296556 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:53.461031 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:53.461002 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:54.272369 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:54.272329 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:54.453921 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:54.453885 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:54.602494 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:54.602406 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:55.990359 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:55.990328 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:50:58.376670 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:50:58.376637 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:01.183035 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:01.182998 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:06.358326 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:06.358279 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:11.381954 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:11.381916 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:16.543968 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:16.543932 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:18.557621 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:51:18.557580 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:51:21.559493 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:21.559442 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:26.669056 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:26.669024 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:31.656744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:31.656700 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:36.757415 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:36.757372 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:41.756588 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:41.756548 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:46.857065 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:46.857027 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:48.557438 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:51:48.557401 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:51:51.865566 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:51.865537 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:52.591474 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:52.591437 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:56.969638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:56.969605 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:51:59.761865 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:51:59.761824 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:00.274987 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:00.274950 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:01.960527 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:01.960484 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:04.835252 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:04.835220 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:06.284023 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:06.283975 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:06.416308 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:06.416273 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:06.547850 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:06.547762 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:06.749811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:06.749778 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:12.405439 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:12.405409 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:12.535919 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:12.535882 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:18.558664 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:52:18.558624 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:52:40.756469 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:40.756430 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:40.955676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:40.955644 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:41.258950 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:41.258916 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:42.157910 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:42.157867 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:42.988451 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:42.988409 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:43.160158 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:43.160122 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:43.374394 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:43.374309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:43.939761 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:43.939721 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:44.089490 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:44.089456 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:47.979569 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:47.979532 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:48.103122 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:48.103087 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:48.267559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:48.267473 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:48.559699 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:52:48.559546 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:52:52.929472 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:52.929437 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:53.350398 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:53.350309 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:55.452839 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.452804 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h"] Nov 03 14:52:55.453261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.453225 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a96c72f6-9a2d-4d91-95c5-c603ed3cda18" containerName="revoke-keys" Nov 03 14:52:55.453261 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.453238 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96c72f6-9a2d-4d91-95c5-c603ed3cda18" containerName="revoke-keys" Nov 03 14:52:55.453343 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.453317 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96c72f6-9a2d-4d91-95c5-c603ed3cda18" containerName="revoke-keys" Nov 03 14:52:55.456461 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.456437 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.505440 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.505406 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:52:55.552160 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.552127 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:52:55.554478 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.554454 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h"] Nov 03 14:52:55.568442 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.568401 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4527\" (UniqueName: \"kubernetes.io/projected/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-kube-api-access-x4527\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.568564 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.568457 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.568564 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.568540 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.669908 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.669872 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.670118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.669916 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.670118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.669987 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4527\" (UniqueName: \"kubernetes.io/projected/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-kube-api-access-x4527\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.670669 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.670637 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.672285 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.672264 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.723859 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.723792 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4527\" (UniqueName: \"kubernetes.io/projected/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-kube-api-access-x4527\") pod \"maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:55.769736 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:55.769700 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:56.137361 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:56.137336 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h"] Nov 03 14:52:56.139620 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:52:56.139589 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff611e52_d984_48bd_bf9a_5fc18e3c2de3.slice/crio-a64c3ac973ac9f3fc36a935b0b017971546066ea1aafc7b3da86b534f6a3ff40 WatchSource:0}: Error finding container a64c3ac973ac9f3fc36a935b0b017971546066ea1aafc7b3da86b534f6a3ff40: Status 404 returned error can't find the container with id a64c3ac973ac9f3fc36a935b0b017971546066ea1aafc7b3da86b534f6a3ff40 Nov 03 14:52:56.485088 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:56.485039 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" event={"ID":"ff611e52-d984-48bd-bf9a-5fc18e3c2de3","Type":"ContainerStarted","Data":"8769d94cd08344108f6228691698f62b75b33806e30bbef3b71f22b23457891a"} Nov 03 14:52:56.485480 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:56.485168 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" event={"ID":"ff611e52-d984-48bd-bf9a-5fc18e3c2de3","Type":"ContainerStarted","Data":"a64c3ac973ac9f3fc36a935b0b017971546066ea1aafc7b3da86b534f6a3ff40"} Nov 03 14:52:56.593456 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:56.593402 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" podStartSLOduration=1.5933892969999999 podStartE2EDuration="1.593389297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:56.592354886 +0000 UTC m=+6509.786758482" watchObservedRunningTime="2026-11-03 14:52:56.593389297 +0000 UTC m=+6509.787792928" Nov 03 14:52:57.491440 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:57.491405 4924 generic.go:356] "Generic (PLEG): container finished" podID="ff611e52-d984-48bd-bf9a-5fc18e3c2de3" containerID="8769d94cd08344108f6228691698f62b75b33806e30bbef3b71f22b23457891a" exitCode=0 Nov 03 14:52:57.491846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:57.491495 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" event={"ID":"ff611e52-d984-48bd-bf9a-5fc18e3c2de3","Type":"ContainerDied","Data":"8769d94cd08344108f6228691698f62b75b33806e30bbef3b71f22b23457891a"} Nov 03 14:52:58.358357 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.358318 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:52:58.634537 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.634500 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:58.698705 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.698669 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-xks-ca\") pod \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " Nov 03 14:52:58.698883 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.698762 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-openshift-ca\") pod \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " Nov 03 14:52:58.698883 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.698800 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-kube-api-access-x4527\" (UniqueName: \"kubernetes.io/projected/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-kube-api-access-x4527\") pod \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\" (UID: \"ff611e52-d984-48bd-bf9a-5fc18e3c2de3\") " Nov 03 14:52:58.699230 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.699202 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-openshift-ca" pod "ff611e52-d984-48bd-bf9a-5fc18e3c2de3" (UID: "ff611e52-d984-48bd-bf9a-5fc18e3c2de3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:52:58.700794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.700772 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-xks-ca" pod "ff611e52-d984-48bd-bf9a-5fc18e3c2de3" (UID: "ff611e52-d984-48bd-bf9a-5fc18e3c2de3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:52:58.700871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.700845 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-kube-api-access-x4527" pod "ff611e52-d984-48bd-bf9a-5fc18e3c2de3" (UID: "ff611e52-d984-48bd-bf9a-5fc18e3c2de3"). InnerVolumeSpecName "kube-api-access-x4527". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:52:58.800528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.800489 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:58.800528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.800522 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4527\" (UniqueName: \"kubernetes.io/projected/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-kube-api-access-x4527\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:58.800528 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:58.800533 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff611e52-d984-48bd-bf9a-5fc18e3c2de3-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:59.503205 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:59.503176 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" Nov 03 14:52:59.503205 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:59.503186 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h" event={"ID":"ff611e52-d984-48bd-bf9a-5fc18e3c2de3","Type":"ContainerDied","Data":"a64c3ac973ac9f3fc36a935b0b017971546066ea1aafc7b3da86b534f6a3ff40"} Nov 03 14:52:59.503449 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:52:59.503235 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a64c3ac973ac9f3fc36a935b0b017971546066ea1aafc7b3da86b534f6a3ff40" Nov 03 14:53:03.454718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:03.454654 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:08.476277 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:08.476194 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:09.547781 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:09.547742 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:09.689846 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:09.689810 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:10.111865 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:10.111824 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:12.937868 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:12.937826 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:14.518955 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:14.518916 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:14.811290 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:14.811212 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:15.160081 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:15.160031 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h"] Nov 03 14:53:15.206217 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:15.206182 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-21ea69-g927h"] Nov 03 14:53:15.484834 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:15.484748 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:16.948963 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:16.948932 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff611e52-d984-48bd-bf9a-5fc18e3c2de3" path="/var/lib/kubelet/pods/ff611e52-d984-48bd-bf9a-5fc18e3c2de3/volumes" Nov 03 14:53:18.558761 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:53:18.558720 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:53:36.595872 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:36.595837 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7"] Nov 03 14:53:36.596454 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:36.596029 4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" podUID="27a5a362-b740-42a2-8f27-c5a918dfba78" containerName="manager" containerID="cri-o://c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb" gracePeriod=10 Nov 03 14:53:37.055778 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.055752 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 14:53:37.180354 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.180324 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a5a362-b740-42a2-8f27-c5a918dfba78-kube-api-access-b4h4x\" (UniqueName: \"kubernetes.io/projected/27a5a362-b740-42a2-8f27-c5a918dfba78-kube-api-access-b4h4x\") pod \"27a5a362-b740-42a2-8f27-c5a918dfba78\" (UID: \"27a5a362-b740-42a2-8f27-c5a918dfba78\") " Nov 03 14:53:37.180559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.180370 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a5a362-b740-42a2-8f27-c5a918dfba78-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a5a362-b740-42a2-8f27-c5a918dfba78-extensions-socket-volume\") pod \"27a5a362-b740-42a2-8f27-c5a918dfba78\" (UID: \"27a5a362-b740-42a2-8f27-c5a918dfba78\") " Nov 03 14:53:37.180879 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.180849 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a5a362-b740-42a2-8f27-c5a918dfba78-extensions-socket-volume" pod "27a5a362-b740-42a2-8f27-c5a918dfba78" (UID: "27a5a362-b740-42a2-8f27-c5a918dfba78"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:53:37.182605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.182575 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a5a362-b740-42a2-8f27-c5a918dfba78-kube-api-access-b4h4x" pod "27a5a362-b740-42a2-8f27-c5a918dfba78" (UID: "27a5a362-b740-42a2-8f27-c5a918dfba78"). InnerVolumeSpecName "kube-api-access-b4h4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:53:37.281758 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.281712 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4h4x\" (UniqueName: \"kubernetes.io/projected/27a5a362-b740-42a2-8f27-c5a918dfba78-kube-api-access-b4h4x\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:37.281758 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.281756 4924 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a5a362-b740-42a2-8f27-c5a918dfba78-extensions-socket-volume\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:37.728997 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.728960 4924 generic.go:356] "Generic (PLEG): container finished" podID="27a5a362-b740-42a2-8f27-c5a918dfba78" containerID="c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb" exitCode=0 Nov 03 14:53:37.729477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.729006 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" Nov 03 14:53:37.729477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.729049 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" event={"ID":"27a5a362-b740-42a2-8f27-c5a918dfba78","Type":"ContainerDied","Data":"c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb"} Nov 03 14:53:37.729477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.729129 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7" event={"ID":"27a5a362-b740-42a2-8f27-c5a918dfba78","Type":"ContainerDied","Data":"28092461ec2dfd86418846b870c06386c316562b8f01a137aa61aa37806ddcda"} Nov 03 14:53:37.729477 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.729154 4924 scope.go:122] "RemoveContainer" containerID="c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb" Nov 03 14:53:37.745149 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.745126 4924 scope.go:122] "RemoveContainer" containerID="c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb" Nov 03 14:53:37.745423 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:53:37.745399 4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb\": container with ID starting with c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb not found: ID does not exist" containerID="c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb" Nov 03 14:53:37.745486 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.745434 4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb"} err="failed to get container status \"c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb\": rpc error: code = NotFound desc = could not find container \"c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb\": container with ID starting with c49ccbfab456100c560d281ff9f641a9776d1a935ddb1ba3f2126f50d06348eb not found: ID does not exist" Nov 03 14:53:37.883878 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.883842 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7"] Nov 03 14:53:37.889399 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:37.889373 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8h2h7"] Nov 03 14:53:38.949696 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:38.949660 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a5a362-b740-42a2-8f27-c5a918dfba78" path="/var/lib/kubelet/pods/27a5a362-b740-42a2-8f27-c5a918dfba78/volumes" Nov 03 14:53:48.557609 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:53:48.557568 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:53:53.270791 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:53.270753 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:57.603138 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:57.603104 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:53:57.716909 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:53:57.716872 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:54:02.708026 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:02.707987 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:54:18.557264 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:54:18.557227 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:54:27.996169 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:27.996127 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:54:28.030370 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:28.030331 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:54:40.197940 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.197855 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl"] Nov 03 14:54:40.198484 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.198461 4924 cpu_manager.go:414] "containerMap: removing container" podUID="ff611e52-d984-48bd-bf9a-5fc18e3c2de3" containerName="revoke-keys" Nov 03 14:54:40.198578 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.198487 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff611e52-d984-48bd-bf9a-5fc18e3c2de3" containerName="revoke-keys" Nov 03 14:54:40.198578 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.198508 4924 cpu_manager.go:414] "containerMap: removing container" podUID="27a5a362-b740-42a2-8f27-c5a918dfba78" containerName="manager" Nov 03 14:54:40.198578 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.198522 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a5a362-b740-42a2-8f27-c5a918dfba78" containerName="manager" Nov 03 14:54:40.198752 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.198668 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff611e52-d984-48bd-bf9a-5fc18e3c2de3" containerName="revoke-keys" Nov 03 14:54:40.198752 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.198684 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a5a362-b740-42a2-8f27-c5a918dfba78" containerName="manager" Nov 03 14:54:40.203108 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.203085 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:40.252733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.252703 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zk57w\"" Nov 03 14:54:40.301627 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.301593 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl"] Nov 03 14:54:40.349876 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.349840 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7gd\" (UniqueName: \"kubernetes.io/projected/a444f2ee-fc8a-4780-8058-3e5e6c2fae73-kube-api-access-5z7gd\") pod \"kuadrant-operator-controller-manager-678d87766f-x2vzl\" (UID: \"a444f2ee-fc8a-4780-8058-3e5e6c2fae73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:40.350066 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.349884 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a444f2ee-fc8a-4780-8058-3e5e6c2fae73-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x2vzl\" (UID: \"a444f2ee-fc8a-4780-8058-3e5e6c2fae73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:40.451321 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.451234 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7gd\" (UniqueName: \"kubernetes.io/projected/a444f2ee-fc8a-4780-8058-3e5e6c2fae73-kube-api-access-5z7gd\") pod \"kuadrant-operator-controller-manager-678d87766f-x2vzl\" (UID: \"a444f2ee-fc8a-4780-8058-3e5e6c2fae73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:40.451321 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.451284 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a444f2ee-fc8a-4780-8058-3e5e6c2fae73-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x2vzl\" (UID: \"a444f2ee-fc8a-4780-8058-3e5e6c2fae73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:40.451650 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.451630 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a444f2ee-fc8a-4780-8058-3e5e6c2fae73-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x2vzl\" (UID: \"a444f2ee-fc8a-4780-8058-3e5e6c2fae73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:40.507333 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.507300 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7gd\" (UniqueName: \"kubernetes.io/projected/a444f2ee-fc8a-4780-8058-3e5e6c2fae73-kube-api-access-5z7gd\") pod \"kuadrant-operator-controller-manager-678d87766f-x2vzl\" (UID: \"a444f2ee-fc8a-4780-8058-3e5e6c2fae73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:40.517211 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.517188 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:40.978347 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:40.978315 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl"] Nov 03 14:54:40.979891 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:54:40.979859 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda444f2ee_fc8a_4780_8058_3e5e6c2fae73.slice/crio-61fd969d45c86ca2ca2b89a206c0d4a644a31118d9532e474706bc57d95dff3a WatchSource:0}: Error finding container 61fd969d45c86ca2ca2b89a206c0d4a644a31118d9532e474706bc57d95dff3a: Status 404 returned error can't find the container with id 61fd969d45c86ca2ca2b89a206c0d4a644a31118d9532e474706bc57d95dff3a Nov 03 14:54:41.068771 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:41.068731 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" event={"ID":"a444f2ee-fc8a-4780-8058-3e5e6c2fae73","Type":"ContainerStarted","Data":"3d818f1b2b5d2835ad826bd2226728ecafae299aed373c09f949a5f1b15b0c2d"} Nov 03 14:54:41.068967 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:41.068796 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" event={"ID":"a444f2ee-fc8a-4780-8058-3e5e6c2fae73","Type":"ContainerStarted","Data":"61fd969d45c86ca2ca2b89a206c0d4a644a31118d9532e474706bc57d95dff3a"} Nov 03 14:54:41.068967 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:41.068877 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:41.197757 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:41.197706 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" podStartSLOduration=1.1976926159999999 podStartE2EDuration="1.197692616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:41.195322638 +0000 UTC m=+6614.389726232" watchObservedRunningTime="2026-11-03 14:54:41.197692616 +0000 UTC m=+6614.392096209" Nov 03 14:54:48.557874 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:54:48.557837 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:54:52.077015 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:52.076983 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x2vzl" Nov 03 14:54:53.600406 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:54:53.600371 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:55:01.364362 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:55:01.364323 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:55:01.549795 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:55:01.549717 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:55:08.934056 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:55:08.934022 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:55:18.559724 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:55:18.559677 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:55:42.679597 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:55:42.679559 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:55:42.800023 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:55:42.799984 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:55:48.558217 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:55:48.558175 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:55:54.091560 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:55:54.091509 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:55:55.904241 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:55:55.904169 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:55:55.904949 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:55:55.904909 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:56:08.107608 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.107530 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt"] Nov 03 14:56:08.111155 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.111134 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.142684 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.142654 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.142831 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.142693 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr5hc\" (UniqueName: \"kubernetes.io/projected/2b526644-eb84-4cae-82bd-350ea2bc6dcc-kube-api-access-qr5hc\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.142907 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.142870 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.160284 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.160254 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:56:08.206717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.206687 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:56:08.209028 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.209003 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt"] Nov 03 14:56:08.244164 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.244138 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr5hc\" (UniqueName: \"kubernetes.io/projected/2b526644-eb84-4cae-82bd-350ea2bc6dcc-kube-api-access-qr5hc\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.244318 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.244299 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.244423 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.244405 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.245019 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.245001 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.246566 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.246548 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.298917 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.298879 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr5hc\" (UniqueName: \"kubernetes.io/projected/2b526644-eb84-4cae-82bd-350ea2bc6dcc-kube-api-access-qr5hc\") pod \"maas-api-revoke-sub-6f6b482d-dsrwt\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.424089 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.424044 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:08.788912 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:08.788885 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt"] Nov 03 14:56:08.790516 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:56:08.790477 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b526644_eb84_4cae_82bd_350ea2bc6dcc.slice/crio-fc2ad93873e04f3752018324d53f7210578a0f63daf3d3659e15ba19881d12a3 WatchSource:0}: Error finding container fc2ad93873e04f3752018324d53f7210578a0f63daf3d3659e15ba19881d12a3: Status 404 returned error can't find the container with id fc2ad93873e04f3752018324d53f7210578a0f63daf3d3659e15ba19881d12a3 Nov 03 14:56:09.679046 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:09.679009 4924 generic.go:356] "Generic (PLEG): container finished" podID="2b526644-eb84-4cae-82bd-350ea2bc6dcc" containerID="ea4aa85a5c4973a72b7972e0baf2fb52f50adec6682c3bcab982ab8c986e2f73" exitCode=0 Nov 03 14:56:09.679457 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:09.679094 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" event={"ID":"2b526644-eb84-4cae-82bd-350ea2bc6dcc","Type":"ContainerDied","Data":"ea4aa85a5c4973a72b7972e0baf2fb52f50adec6682c3bcab982ab8c986e2f73"} Nov 03 14:56:09.679457 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:09.679169 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" event={"ID":"2b526644-eb84-4cae-82bd-350ea2bc6dcc","Type":"ContainerStarted","Data":"fc2ad93873e04f3752018324d53f7210578a0f63daf3d3659e15ba19881d12a3"} Nov 03 14:56:10.817403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.817376 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:10.869818 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.869787 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-openshift-ca\") pod \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " Nov 03 14:56:10.870004 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.869868 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b526644-eb84-4cae-82bd-350ea2bc6dcc-kube-api-access-qr5hc\" (UniqueName: \"kubernetes.io/projected/2b526644-eb84-4cae-82bd-350ea2bc6dcc-kube-api-access-qr5hc\") pod \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " Nov 03 14:56:10.870004 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.869908 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-xks-ca\") pod \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\" (UID: \"2b526644-eb84-4cae-82bd-350ea2bc6dcc\") " Nov 03 14:56:10.870212 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.870188 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-openshift-ca" pod "2b526644-eb84-4cae-82bd-350ea2bc6dcc" (UID: "2b526644-eb84-4cae-82bd-350ea2bc6dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:56:10.872003 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.871974 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-xks-ca" pod "2b526644-eb84-4cae-82bd-350ea2bc6dcc" (UID: "2b526644-eb84-4cae-82bd-350ea2bc6dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:56:10.872154 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.872133 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b526644-eb84-4cae-82bd-350ea2bc6dcc-kube-api-access-qr5hc" pod "2b526644-eb84-4cae-82bd-350ea2bc6dcc" (UID: "2b526644-eb84-4cae-82bd-350ea2bc6dcc"). InnerVolumeSpecName "kube-api-access-qr5hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:56:10.971497 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.971468 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr5hc\" (UniqueName: \"kubernetes.io/projected/2b526644-eb84-4cae-82bd-350ea2bc6dcc-kube-api-access-qr5hc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:10.971497 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.971495 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:10.971685 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:10.971506 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b526644-eb84-4cae-82bd-350ea2bc6dcc-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:11.692038 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:11.692006 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" Nov 03 14:56:11.692252 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:11.692048 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt" event={"ID":"2b526644-eb84-4cae-82bd-350ea2bc6dcc","Type":"ContainerDied","Data":"fc2ad93873e04f3752018324d53f7210578a0f63daf3d3659e15ba19881d12a3"} Nov 03 14:56:11.692252 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:11.692128 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2ad93873e04f3752018324d53f7210578a0f63daf3d3659e15ba19881d12a3" Nov 03 14:56:18.557453 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:56:18.557417 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:56:18.918896 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:18.918863 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr"] Nov 03 14:56:18.919342 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:18.919327 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2b526644-eb84-4cae-82bd-350ea2bc6dcc" containerName="revoke-keys" Nov 03 14:56:18.919385 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:18.919343 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b526644-eb84-4cae-82bd-350ea2bc6dcc" containerName="revoke-keys" Nov 03 14:56:18.919425 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:18.919416 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b526644-eb84-4cae-82bd-350ea2bc6dcc" containerName="revoke-keys" Nov 03 14:56:18.922180 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:18.922154 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.017266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.017231 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:56:19.017266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.017232 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:56:19.019934 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.019904 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr"] Nov 03 14:56:19.047979 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.047949 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.048181 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.047993 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz958\" (UniqueName: \"kubernetes.io/projected/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-kube-api-access-xz958\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.048181 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.048096 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.149230 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.149188 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.149406 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.149310 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.149406 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.149354 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz958\" (UniqueName: \"kubernetes.io/projected/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-kube-api-access-xz958\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.150050 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.150027 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.151614 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.151589 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.205614 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.205547 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz958\" (UniqueName: \"kubernetes.io/projected/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-kube-api-access-xz958\") pod \"maas-api-revoke-sub-4df84cea-dktzr\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.234063 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.234028 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:19.600535 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.600508 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr"] Nov 03 14:56:19.602123 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:56:19.602051 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f11bfb_85a4_44b6_8018_a3c8b0cde046.slice/crio-bb8e5dacbfb5f96553a00c62e09af00c849ee2e61bb37393c365b588e458591a WatchSource:0}: Error finding container bb8e5dacbfb5f96553a00c62e09af00c849ee2e61bb37393c365b588e458591a: Status 404 returned error can't find the container with id bb8e5dacbfb5f96553a00c62e09af00c849ee2e61bb37393c365b588e458591a Nov 03 14:56:19.735929 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:19.735893 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" event={"ID":"a7f11bfb-85a4-44b6-8018-a3c8b0cde046","Type":"ContainerStarted","Data":"bb8e5dacbfb5f96553a00c62e09af00c849ee2e61bb37393c365b588e458591a"} Nov 03 14:56:20.742733 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:20.742653 4924 generic.go:356] "Generic (PLEG): container finished" podID="a7f11bfb-85a4-44b6-8018-a3c8b0cde046" containerID="b11adaffa08931d8da3b8caefcb73042076703ff53aa7e3f3d30f3487e37006b" exitCode=0 Nov 03 14:56:20.743129 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:20.742735 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" event={"ID":"a7f11bfb-85a4-44b6-8018-a3c8b0cde046","Type":"ContainerDied","Data":"b11adaffa08931d8da3b8caefcb73042076703ff53aa7e3f3d30f3487e37006b"} Nov 03 14:56:21.886845 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:21.886820 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:21.983665 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:21.983627 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-kube-api-access-xz958\" (UniqueName: \"kubernetes.io/projected/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-kube-api-access-xz958\") pod \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " Nov 03 14:56:21.983894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:21.983720 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-openshift-ca\") pod \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " Nov 03 14:56:21.983894 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:21.983746 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-xks-ca\") pod \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\" (UID: \"a7f11bfb-85a4-44b6-8018-a3c8b0cde046\") " Nov 03 14:56:21.984734 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:21.984686 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-openshift-ca" pod "a7f11bfb-85a4-44b6-8018-a3c8b0cde046" (UID: "a7f11bfb-85a4-44b6-8018-a3c8b0cde046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:56:21.986408 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:21.986385 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-xks-ca" pod "a7f11bfb-85a4-44b6-8018-a3c8b0cde046" (UID: "a7f11bfb-85a4-44b6-8018-a3c8b0cde046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:56:21.987021 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:21.987003 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-kube-api-access-xz958" pod "a7f11bfb-85a4-44b6-8018-a3c8b0cde046" (UID: "a7f11bfb-85a4-44b6-8018-a3c8b0cde046"). InnerVolumeSpecName "kube-api-access-xz958". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:56:22.085610 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:22.085531 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz958\" (UniqueName: \"kubernetes.io/projected/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-kube-api-access-xz958\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:22.085610 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:22.085559 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:22.085610 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:22.085570 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f11bfb-85a4-44b6-8018-a3c8b0cde046-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:22.755851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:22.755819 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" Nov 03 14:56:22.756065 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:22.755853 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr" event={"ID":"a7f11bfb-85a4-44b6-8018-a3c8b0cde046","Type":"ContainerDied","Data":"bb8e5dacbfb5f96553a00c62e09af00c849ee2e61bb37393c365b588e458591a"} Nov 03 14:56:22.756065 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:22.755902 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb8e5dacbfb5f96553a00c62e09af00c849ee2e61bb37393c365b588e458591a" Nov 03 14:56:48.557740 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:56:48.557697 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:56:52.548303 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.548260 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h"] Nov 03 14:56:52.549118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.549091 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a7f11bfb-85a4-44b6-8018-a3c8b0cde046" containerName="revoke-keys" Nov 03 14:56:52.549118 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.549117 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f11bfb-85a4-44b6-8018-a3c8b0cde046" containerName="revoke-keys" Nov 03 14:56:52.549288 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.549258 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f11bfb-85a4-44b6-8018-a3c8b0cde046" containerName="revoke-keys" Nov 03 14:56:52.552715 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.552694 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.601225 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.601187 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.601225 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.601227 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzw7\" (UniqueName: \"kubernetes.io/projected/2e82ee7a-f7a5-4af9-86a0-266c30de9330-kube-api-access-6gzw7\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.601479 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.601326 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.648342 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.648300 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:56:52.648523 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.648346 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:56:52.651017 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.650990 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h"] Nov 03 14:56:52.702425 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.702388 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.702626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.702480 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.702626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.702502 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzw7\" (UniqueName: \"kubernetes.io/projected/2e82ee7a-f7a5-4af9-86a0-266c30de9330-kube-api-access-6gzw7\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.703266 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.703243 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.704983 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.704954 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.756274 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.756238 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzw7\" (UniqueName: \"kubernetes.io/projected/2e82ee7a-f7a5-4af9-86a0-266c30de9330-kube-api-access-6gzw7\") pod \"maas-api-revoke-sub-1fe9d177-4qb5h\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:52.866048 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:52.865957 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:53.230552 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:53.230522 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h"] Nov 03 14:56:53.232156 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:56:53.232117 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e82ee7a_f7a5_4af9_86a0_266c30de9330.slice/crio-f74958ff63724722ea83e8def7ab7dc7157fe7a644f6373283685cbf62e25bab WatchSource:0}: Error finding container f74958ff63724722ea83e8def7ab7dc7157fe7a644f6373283685cbf62e25bab: Status 404 returned error can't find the container with id f74958ff63724722ea83e8def7ab7dc7157fe7a644f6373283685cbf62e25bab Nov 03 14:56:53.948691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:53.948653 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" event={"ID":"2e82ee7a-f7a5-4af9-86a0-266c30de9330","Type":"ContainerStarted","Data":"4998198d37e6df9b4d1267e8048d2a1a6b3e03fd659890e4536df81eebd56fde"} Nov 03 14:56:53.949139 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:53.948716 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" event={"ID":"2e82ee7a-f7a5-4af9-86a0-266c30de9330","Type":"ContainerStarted","Data":"f74958ff63724722ea83e8def7ab7dc7157fe7a644f6373283685cbf62e25bab"} Nov 03 14:56:54.058181 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:54.058128 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" podStartSLOduration=2.058054958 podStartE2EDuration="2.058054958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:56:54.057430272 +0000 UTC m=+6747.251833866" watchObservedRunningTime="2026-11-03 14:56:54.058054958 +0000 UTC m=+6747.252458551" Nov 03 14:56:54.955348 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:54.955309 4924 generic.go:356] "Generic (PLEG): container finished" podID="2e82ee7a-f7a5-4af9-86a0-266c30de9330" containerID="4998198d37e6df9b4d1267e8048d2a1a6b3e03fd659890e4536df81eebd56fde" exitCode=0 Nov 03 14:56:54.955776 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:54.955390 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" event={"ID":"2e82ee7a-f7a5-4af9-86a0-266c30de9330","Type":"ContainerDied","Data":"4998198d37e6df9b4d1267e8048d2a1a6b3e03fd659890e4536df81eebd56fde"} Nov 03 14:56:55.304240 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:55.304146 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:56:56.099590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.099557 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:56.134541 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.134508 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-openshift-ca\") pod \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " Nov 03 14:56:56.134720 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.134595 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e82ee7a-f7a5-4af9-86a0-266c30de9330-kube-api-access-6gzw7\" (UniqueName: \"kubernetes.io/projected/2e82ee7a-f7a5-4af9-86a0-266c30de9330-kube-api-access-6gzw7\") pod \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " Nov 03 14:56:56.134720 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.134658 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-xks-ca\") pod \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\" (UID: \"2e82ee7a-f7a5-4af9-86a0-266c30de9330\") " Nov 03 14:56:56.134873 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.134850 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-openshift-ca" pod "2e82ee7a-f7a5-4af9-86a0-266c30de9330" (UID: "2e82ee7a-f7a5-4af9-86a0-266c30de9330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:56:56.135006 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.134991 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:56.136637 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.136610 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e82ee7a-f7a5-4af9-86a0-266c30de9330-kube-api-access-6gzw7" pod "2e82ee7a-f7a5-4af9-86a0-266c30de9330" (UID: "2e82ee7a-f7a5-4af9-86a0-266c30de9330"). InnerVolumeSpecName "kube-api-access-6gzw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:56:56.136637 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.136623 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-xks-ca" pod "2e82ee7a-f7a5-4af9-86a0-266c30de9330" (UID: "2e82ee7a-f7a5-4af9-86a0-266c30de9330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:56:56.236226 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.236146 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gzw7\" (UniqueName: \"kubernetes.io/projected/2e82ee7a-f7a5-4af9-86a0-266c30de9330-kube-api-access-6gzw7\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:56.236226 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.236187 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e82ee7a-f7a5-4af9-86a0-266c30de9330-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:56:56.967364 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.967340 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" Nov 03 14:56:56.967511 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.967376 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h" event={"ID":"2e82ee7a-f7a5-4af9-86a0-266c30de9330","Type":"ContainerDied","Data":"f74958ff63724722ea83e8def7ab7dc7157fe7a644f6373283685cbf62e25bab"} Nov 03 14:56:56.967511 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:56:56.967420 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f74958ff63724722ea83e8def7ab7dc7157fe7a644f6373283685cbf62e25bab" Nov 03 14:57:14.390008 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:14.389961 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:57:14.390568 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:14.390104 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:57:18.559913 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:57:18.559878 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:57:33.035950 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:33.035897 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:57:37.253674 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.253590 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t"] Nov 03 14:57:37.254065 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.254047 4924 cpu_manager.go:414] "containerMap: removing container" podUID="2e82ee7a-f7a5-4af9-86a0-266c30de9330" containerName="revoke-keys" Nov 03 14:57:37.254140 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.254066 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e82ee7a-f7a5-4af9-86a0-266c30de9330" containerName="revoke-keys" Nov 03 14:57:37.254226 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.254214 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e82ee7a-f7a5-4af9-86a0-266c30de9330" containerName="revoke-keys" Nov 03 14:57:37.257742 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.257716 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.329520 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.329478 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xwz\" (UniqueName: \"kubernetes.io/projected/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-kube-api-access-t5xwz\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.329717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.329604 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.329717 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.329689 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.353257 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.353227 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:57:37.353409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.353314 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:57:37.355893 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.355872 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t"] Nov 03 14:57:37.430254 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.430217 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.430430 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.430276 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xwz\" (UniqueName: \"kubernetes.io/projected/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-kube-api-access-t5xwz\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.430486 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.430444 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.431098 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.431051 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.432718 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.432697 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.484629 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.484594 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xwz\" (UniqueName: \"kubernetes.io/projected/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-kube-api-access-t5xwz\") pod \"maas-api-revoke-sub-8c40aa77-bld6t\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.573882 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.573798 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:37.941755 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:37.941730 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t"] Nov 03 14:57:37.943003 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:57:37.942972 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32c63dd_a70d_43ee_a5a0_126c2c84b33d.slice/crio-591d87975e58f159c7935c4ba311920dd262e344feb82e05f040a442d68a9d3b WatchSource:0}: Error finding container 591d87975e58f159c7935c4ba311920dd262e344feb82e05f040a442d68a9d3b: Status 404 returned error can't find the container with id 591d87975e58f159c7935c4ba311920dd262e344feb82e05f040a442d68a9d3b Nov 03 14:57:38.277624 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:38.277586 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" event={"ID":"a32c63dd-a70d-43ee-a5a0-126c2c84b33d","Type":"ContainerStarted","Data":"ed91266e67e058ad135a0c11970e1188541f9132b26ba32c28f4828b77e94425"} Nov 03 14:57:38.278000 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:38.277647 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" event={"ID":"a32c63dd-a70d-43ee-a5a0-126c2c84b33d","Type":"ContainerStarted","Data":"591d87975e58f159c7935c4ba311920dd262e344feb82e05f040a442d68a9d3b"} Nov 03 14:57:38.387468 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:38.387409 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" podStartSLOduration=1.387395471 podStartE2EDuration="1.387395471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:57:38.385210357 +0000 UTC m=+6791.579613950" watchObservedRunningTime="2026-11-03 14:57:38.387395471 +0000 UTC m=+6791.581799065" Nov 03 14:57:39.284000 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:39.283961 4924 generic.go:356] "Generic (PLEG): container finished" podID="a32c63dd-a70d-43ee-a5a0-126c2c84b33d" containerID="ed91266e67e058ad135a0c11970e1188541f9132b26ba32c28f4828b77e94425" exitCode=0 Nov 03 14:57:39.284422 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:39.284046 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" event={"ID":"a32c63dd-a70d-43ee-a5a0-126c2c84b33d","Type":"ContainerDied","Data":"ed91266e67e058ad135a0c11970e1188541f9132b26ba32c28f4828b77e94425"} Nov 03 14:57:40.437758 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.437726 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:40.560469 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.560390 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-xks-ca\") pod \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " Nov 03 14:57:40.560469 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.560459 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-openshift-ca\") pod \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " Nov 03 14:57:40.560729 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.560508 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-kube-api-access-t5xwz\" (UniqueName: \"kubernetes.io/projected/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-kube-api-access-t5xwz\") pod \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\" (UID: \"a32c63dd-a70d-43ee-a5a0-126c2c84b33d\") " Nov 03 14:57:40.560859 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.560827 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-openshift-ca" pod "a32c63dd-a70d-43ee-a5a0-126c2c84b33d" (UID: "a32c63dd-a70d-43ee-a5a0-126c2c84b33d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:57:40.562609 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.562585 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-kube-api-access-t5xwz" pod "a32c63dd-a70d-43ee-a5a0-126c2c84b33d" (UID: "a32c63dd-a70d-43ee-a5a0-126c2c84b33d"). InnerVolumeSpecName "kube-api-access-t5xwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:57:40.562713 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.562603 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-xks-ca" pod "a32c63dd-a70d-43ee-a5a0-126c2c84b33d" (UID: "a32c63dd-a70d-43ee-a5a0-126c2c84b33d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:57:40.662049 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.662014 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5xwz\" (UniqueName: \"kubernetes.io/projected/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-kube-api-access-t5xwz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:57:40.662049 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.662046 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:57:40.662049 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:40.662056 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32c63dd-a70d-43ee-a5a0-126c2c84b33d-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:57:41.298976 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:41.298895 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" event={"ID":"a32c63dd-a70d-43ee-a5a0-126c2c84b33d","Type":"ContainerDied","Data":"591d87975e58f159c7935c4ba311920dd262e344feb82e05f040a442d68a9d3b"} Nov 03 14:57:41.298976 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:41.298925 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t" Nov 03 14:57:41.298976 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:41.298948 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="591d87975e58f159c7935c4ba311920dd262e344feb82e05f040a442d68a9d3b" Nov 03 14:57:48.009370 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.009331 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9"] Nov 03 14:57:48.009836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.009816 4924 cpu_manager.go:414] "containerMap: removing container" podUID="a32c63dd-a70d-43ee-a5a0-126c2c84b33d" containerName="revoke-keys" Nov 03 14:57:48.009836 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.009830 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32c63dd-a70d-43ee-a5a0-126c2c84b33d" containerName="revoke-keys" Nov 03 14:57:48.009950 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.009938 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32c63dd-a70d-43ee-a5a0-126c2c84b33d" containerName="revoke-keys" Nov 03 14:57:48.018270 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.018244 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.070193 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.070158 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9"] Nov 03 14:57:48.114013 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.113976 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:57:48.114189 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.114087 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:57:48.126607 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.126577 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76zw\" (UniqueName: \"kubernetes.io/projected/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-kube-api-access-j76zw\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.126729 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.126623 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.126801 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.126729 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.228062 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.228025 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76zw\" (UniqueName: \"kubernetes.io/projected/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-kube-api-access-j76zw\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.228258 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.228096 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.228258 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.228150 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.228799 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.228772 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.230465 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.230440 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.282106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.282024 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76zw\" (UniqueName: \"kubernetes.io/projected/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-kube-api-access-j76zw\") pod \"maas-api-revoke-sub-aeb9e5aa-qn4m9\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.334467 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.334438 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:48.557786 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:57:48.557709 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:57:48.697754 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:48.697490 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9"] Nov 03 14:57:48.699484 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:57:48.699451 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79d57cc_9fac_4ee6_9c2c_a07cad9060cc.slice/crio-cd89214fde1a04b134493e90c6da1a27e581600a3c94d0823d89fd2f22db03ab WatchSource:0}: Error finding container cd89214fde1a04b134493e90c6da1a27e581600a3c94d0823d89fd2f22db03ab: Status 404 returned error can't find the container with id cd89214fde1a04b134493e90c6da1a27e581600a3c94d0823d89fd2f22db03ab Nov 03 14:57:49.344811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:49.344727 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" event={"ID":"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc","Type":"ContainerStarted","Data":"17674b5146cfbea9db5da56f1342aae3068bf98253fb3ecb60541474151898dd"} Nov 03 14:57:49.344811 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:49.344787 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" event={"ID":"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc","Type":"ContainerStarted","Data":"cd89214fde1a04b134493e90c6da1a27e581600a3c94d0823d89fd2f22db03ab"} Nov 03 14:57:49.456323 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:49.456249 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" podStartSLOduration=2.45622941 podStartE2EDuration="2.45622941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:57:49.453430885 +0000 UTC m=+6802.647834480" watchObservedRunningTime="2026-11-03 14:57:49.45622941 +0000 UTC m=+6802.650633004" Nov 03 14:57:50.352824 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:50.352783 4924 generic.go:356] "Generic (PLEG): container finished" podID="e79d57cc-9fac-4ee6-9c2c-a07cad9060cc" containerID="17674b5146cfbea9db5da56f1342aae3068bf98253fb3ecb60541474151898dd" exitCode=0 Nov 03 14:57:50.353374 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:50.352869 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" event={"ID":"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc","Type":"ContainerDied","Data":"17674b5146cfbea9db5da56f1342aae3068bf98253fb3ecb60541474151898dd"} Nov 03 14:57:51.505207 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.505175 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:51.557101 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.557040 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-xks-ca\") pod \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " Nov 03 14:57:51.557101 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.557105 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-kube-api-access-j76zw\" (UniqueName: \"kubernetes.io/projected/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-kube-api-access-j76zw\") pod \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " Nov 03 14:57:51.557362 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.557165 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-openshift-ca\") pod \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\" (UID: \"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc\") " Nov 03 14:57:51.557555 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.557533 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-openshift-ca" pod "e79d57cc-9fac-4ee6-9c2c-a07cad9060cc" (UID: "e79d57cc-9fac-4ee6-9c2c-a07cad9060cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:57:51.559364 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.559336 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-xks-ca" pod "e79d57cc-9fac-4ee6-9c2c-a07cad9060cc" (UID: "e79d57cc-9fac-4ee6-9c2c-a07cad9060cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:57:51.559498 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.559479 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-kube-api-access-j76zw" pod "e79d57cc-9fac-4ee6-9c2c-a07cad9060cc" (UID: "e79d57cc-9fac-4ee6-9c2c-a07cad9060cc"). InnerVolumeSpecName "kube-api-access-j76zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:57:51.658106 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.658081 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:57:51.658258 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.658108 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j76zw\" (UniqueName: \"kubernetes.io/projected/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-kube-api-access-j76zw\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:57:51.658258 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:51.658118 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:57:52.367789 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:52.367758 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" Nov 03 14:57:52.367970 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:52.367783 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9" event={"ID":"e79d57cc-9fac-4ee6-9c2c-a07cad9060cc","Type":"ContainerDied","Data":"cd89214fde1a04b134493e90c6da1a27e581600a3c94d0823d89fd2f22db03ab"} Nov 03 14:57:52.367970 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:52.367836 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd89214fde1a04b134493e90c6da1a27e581600a3c94d0823d89fd2f22db03ab" Nov 03 14:57:54.616988 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:57:54.616954 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:58:18.557937 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:58:18.557890 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:58:35.889628 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:35.889587 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2"] Nov 03 14:58:35.890146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:35.890010 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e79d57cc-9fac-4ee6-9c2c-a07cad9060cc" containerName="revoke-keys" Nov 03 14:58:35.890146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:35.890022 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79d57cc-9fac-4ee6-9c2c-a07cad9060cc" containerName="revoke-keys" Nov 03 14:58:35.890146 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:35.890115 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79d57cc-9fac-4ee6-9c2c-a07cad9060cc" containerName="revoke-keys" Nov 03 14:58:35.893522 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:35.893502 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:35.942605 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:35.942574 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:58:35.989411 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:35.989379 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:58:35.992213 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:35.992185 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2"] Nov 03 14:58:36.011781 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.011751 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8q2k\" (UniqueName: \"kubernetes.io/projected/e12ea4bc-a26d-4e82-963a-767e53a3e27f-kube-api-access-p8q2k\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.011909 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.011813 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.011952 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.011914 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.112603 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.112566 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.112765 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.112614 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.112765 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.112662 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8q2k\" (UniqueName: \"kubernetes.io/projected/e12ea4bc-a26d-4e82-963a-767e53a3e27f-kube-api-access-p8q2k\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.113413 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.113383 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.115032 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.115013 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.167638 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.167596 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8q2k\" (UniqueName: \"kubernetes.io/projected/e12ea4bc-a26d-4e82-963a-767e53a3e27f-kube-api-access-p8q2k\") pod \"maas-api-revoke-sub-4155d965-vpwm2\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.206446 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.206404 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:36.572161 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.572136 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2"] Nov 03 14:58:36.573618 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:58:36.573586 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12ea4bc_a26d_4e82_963a_767e53a3e27f.slice/crio-6fd1ac732960d07d6ee8b8ce19a9b748b2d3fa65b59a9ca20c36062e213b42a0 WatchSource:0}: Error finding container 6fd1ac732960d07d6ee8b8ce19a9b748b2d3fa65b59a9ca20c36062e213b42a0: Status 404 returned error can't find the container with id 6fd1ac732960d07d6ee8b8ce19a9b748b2d3fa65b59a9ca20c36062e213b42a0 Nov 03 14:58:36.714133 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:36.714090 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" event={"ID":"e12ea4bc-a26d-4e82-963a-767e53a3e27f","Type":"ContainerStarted","Data":"6fd1ac732960d07d6ee8b8ce19a9b748b2d3fa65b59a9ca20c36062e213b42a0"} Nov 03 14:58:37.724310 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:37.724152 4924 generic.go:356] "Generic (PLEG): container finished" podID="e12ea4bc-a26d-4e82-963a-767e53a3e27f" containerID="ffb00591a90e0bf5833503750268835c6de55ec5db817a3a76f9789b7e3c9155" exitCode=0 Nov 03 14:58:37.724310 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:37.724249 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" event={"ID":"e12ea4bc-a26d-4e82-963a-767e53a3e27f","Type":"ContainerDied","Data":"ffb00591a90e0bf5833503750268835c6de55ec5db817a3a76f9789b7e3c9155"} Nov 03 14:58:38.872018 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:38.871993 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:38.939218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:38.939181 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12ea4bc-a26d-4e82-963a-767e53a3e27f-kube-api-access-p8q2k\" (UniqueName: \"kubernetes.io/projected/e12ea4bc-a26d-4e82-963a-767e53a3e27f-kube-api-access-p8q2k\") pod \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " Nov 03 14:58:38.939408 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:38.939241 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-xks-ca\") pod \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " Nov 03 14:58:38.939408 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:38.939287 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-openshift-ca\") pod \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\" (UID: \"e12ea4bc-a26d-4e82-963a-767e53a3e27f\") " Nov 03 14:58:38.939744 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:38.939699 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-openshift-ca" pod "e12ea4bc-a26d-4e82-963a-767e53a3e27f" (UID: "e12ea4bc-a26d-4e82-963a-767e53a3e27f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:58:38.941403 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:38.941381 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-xks-ca" pod "e12ea4bc-a26d-4e82-963a-767e53a3e27f" (UID: "e12ea4bc-a26d-4e82-963a-767e53a3e27f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:58:38.941479 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:38.941379 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12ea4bc-a26d-4e82-963a-767e53a3e27f-kube-api-access-p8q2k" pod "e12ea4bc-a26d-4e82-963a-767e53a3e27f" (UID: "e12ea4bc-a26d-4e82-963a-767e53a3e27f"). InnerVolumeSpecName "kube-api-access-p8q2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:58:39.040686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:39.040610 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:58:39.040686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:39.040640 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8q2k\" (UniqueName: \"kubernetes.io/projected/e12ea4bc-a26d-4e82-963a-767e53a3e27f-kube-api-access-p8q2k\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:58:39.040686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:39.040655 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ea4bc-a26d-4e82-963a-767e53a3e27f-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:58:39.736704 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:39.736665 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" event={"ID":"e12ea4bc-a26d-4e82-963a-767e53a3e27f","Type":"ContainerDied","Data":"6fd1ac732960d07d6ee8b8ce19a9b748b2d3fa65b59a9ca20c36062e213b42a0"} Nov 03 14:58:39.736883 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:39.736717 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd1ac732960d07d6ee8b8ce19a9b748b2d3fa65b59a9ca20c36062e213b42a0" Nov 03 14:58:39.736883 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:39.736677 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4155d965-vpwm2" Nov 03 14:58:46.634514 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.634476 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp"] Nov 03 14:58:46.634960 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.634942 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e12ea4bc-a26d-4e82-963a-767e53a3e27f" containerName="revoke-keys" Nov 03 14:58:46.635008 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.634961 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12ea4bc-a26d-4e82-963a-767e53a3e27f" containerName="revoke-keys" Nov 03 14:58:46.635053 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.635034 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12ea4bc-a26d-4e82-963a-767e53a3e27f" containerName="revoke-keys" Nov 03 14:58:46.638364 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.638343 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.729626 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.729589 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:58:46.729830 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.729678 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:58:46.736671 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.736643 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp"] Nov 03 14:58:46.814255 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.814218 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.814435 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.814314 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmw8\" (UniqueName: \"kubernetes.io/projected/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-kube-api-access-rdmw8\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.814435 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.814414 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.915683 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.915648 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmw8\" (UniqueName: \"kubernetes.io/projected/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-kube-api-access-rdmw8\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.915851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.915705 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.915851 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.915768 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.916484 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.916463 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.918023 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.918004 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:46.969686 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:46.969656 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmw8\" (UniqueName: \"kubernetes.io/projected/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-kube-api-access-rdmw8\") pod \"maas-api-revoke-sub-d4e2751f-jm7hp\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:47.253662 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:47.253579 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:47.622768 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:47.622736 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp"] Nov 03 14:58:47.624522 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:58:47.624492 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354b0c41_d41a_4fc6_83c4_9b7beaf3659d.slice/crio-238ba075154c31646a59e51c43eccc5725c60490b6e0b9f74ff29e23c7c2c31c WatchSource:0}: Error finding container 238ba075154c31646a59e51c43eccc5725c60490b6e0b9f74ff29e23c7c2c31c: Status 404 returned error can't find the container with id 238ba075154c31646a59e51c43eccc5725c60490b6e0b9f74ff29e23c7c2c31c Nov 03 14:58:47.788556 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:47.788514 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" event={"ID":"354b0c41-d41a-4fc6-83c4-9b7beaf3659d","Type":"ContainerStarted","Data":"238ba075154c31646a59e51c43eccc5725c60490b6e0b9f74ff29e23c7c2c31c"} Nov 03 14:58:48.558670 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:58:48.558632 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:58:48.797493 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:48.797404 4924 generic.go:356] "Generic (PLEG): container finished" podID="354b0c41-d41a-4fc6-83c4-9b7beaf3659d" containerID="5ba18fd5d7e06a35e031f3250d481c37d0c18793ba0dfd880b3987c3316953fa" exitCode=0 Nov 03 14:58:48.797872 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:48.797494 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" event={"ID":"354b0c41-d41a-4fc6-83c4-9b7beaf3659d","Type":"ContainerDied","Data":"5ba18fd5d7e06a35e031f3250d481c37d0c18793ba0dfd880b3987c3316953fa"} Nov 03 14:58:50.105591 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.105506 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:50.258590 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.258526 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-kube-api-access-rdmw8\" (UniqueName: \"kubernetes.io/projected/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-kube-api-access-rdmw8\") pod \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " Nov 03 14:58:50.258941 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.258917 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-xks-ca\") pod \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " Nov 03 14:58:50.259188 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.259167 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-openshift-ca\") pod \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\" (UID: \"354b0c41-d41a-4fc6-83c4-9b7beaf3659d\") " Nov 03 14:58:50.260185 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.260151 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-openshift-ca" pod "354b0c41-d41a-4fc6-83c4-9b7beaf3659d" (UID: "354b0c41-d41a-4fc6-83c4-9b7beaf3659d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:58:50.313170 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.302725 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-kube-api-access-rdmw8" pod "354b0c41-d41a-4fc6-83c4-9b7beaf3659d" (UID: "354b0c41-d41a-4fc6-83c4-9b7beaf3659d"). InnerVolumeSpecName "kube-api-access-rdmw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:58:50.359316 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.359204 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-xks-ca" pod "354b0c41-d41a-4fc6-83c4-9b7beaf3659d" (UID: "354b0c41-d41a-4fc6-83c4-9b7beaf3659d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:58:50.360683 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.360653 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:58:50.360927 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.360909 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:58:50.361037 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.361025 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdmw8\" (UniqueName: \"kubernetes.io/projected/354b0c41-d41a-4fc6-83c4-9b7beaf3659d-kube-api-access-rdmw8\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:58:50.812720 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.812685 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" Nov 03 14:58:50.812917 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.812716 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e2751f-jm7hp" event={"ID":"354b0c41-d41a-4fc6-83c4-9b7beaf3659d","Type":"ContainerDied","Data":"238ba075154c31646a59e51c43eccc5725c60490b6e0b9f74ff29e23c7c2c31c"} Nov 03 14:58:50.812917 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:50.812800 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="238ba075154c31646a59e51c43eccc5725c60490b6e0b9f74ff29e23c7c2c31c" Nov 03 14:58:54.952628 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:58:54.952595 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:59:07.619126 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:07.619026 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:59:18.558686 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:59:18.558645 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:59:28.045363 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:28.045333 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:59:28.074889 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:28.074862 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2pfz7_50d1326a-e89d-4e45-90c6-9a254a157e48/console-operator/2.log" Nov 03 14:59:29.090794 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:29.090762 4924 scope.go:122] "RemoveContainer" containerID="8769d94cd08344108f6228691698f62b75b33806e30bbef3b71f22b23457891a" Nov 03 14:59:38.252651 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.252612 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg"] Nov 03 14:59:38.253057 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.253045 4924 cpu_manager.go:414] "containerMap: removing container" podUID="354b0c41-d41a-4fc6-83c4-9b7beaf3659d" containerName="revoke-keys" Nov 03 14:59:38.253134 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.253059 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354b0c41-d41a-4fc6-83c4-9b7beaf3659d" containerName="revoke-keys" Nov 03 14:59:38.253173 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.253140 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="354b0c41-d41a-4fc6-83c4-9b7beaf3659d" containerName="revoke-keys" Nov 03 14:59:38.257667 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.257641 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.348165 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.348124 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.348361 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.348180 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.348361 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.348276 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdmj\" (UniqueName: \"kubernetes.io/projected/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-kube-api-access-fqdmj\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.353115 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.353058 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:59:38.353304 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.353206 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:59:38.356115 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.356095 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg"] Nov 03 14:59:38.449218 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.449174 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.449538 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.449237 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.449538 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.449267 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdmj\" (UniqueName: \"kubernetes.io/projected/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-kube-api-access-fqdmj\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.449932 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.449908 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.451614 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.451592 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.503907 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.503833 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdmj\" (UniqueName: \"kubernetes.io/projected/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-kube-api-access-fqdmj\") pod \"maas-api-revoke-sub-5071e830-qjkfg\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.571305 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.571269 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:38.935691 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:38.935666 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg"] Nov 03 14:59:38.937283 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:59:38.937252 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4782f6b_4f8e_40f7_a4ae_d9f84fed5c72.slice/crio-de7dd48c7ebe79148d658db9a26d3bda8a424e7adb49583ae72a9cb32a549261 WatchSource:0}: Error finding container de7dd48c7ebe79148d658db9a26d3bda8a424e7adb49583ae72a9cb32a549261: Status 404 returned error can't find the container with id de7dd48c7ebe79148d658db9a26d3bda8a424e7adb49583ae72a9cb32a549261 Nov 03 14:59:39.232053 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:39.232017 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" event={"ID":"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72","Type":"ContainerStarted","Data":"de7dd48c7ebe79148d658db9a26d3bda8a424e7adb49583ae72a9cb32a549261"} Nov 03 14:59:40.240230 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:40.240186 4924 generic.go:356] "Generic (PLEG): container finished" podID="c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72" containerID="66e6ed119d61905409ae68d61ad8d0061ab8d8b5e83ebe9931438ec7112e66c4" exitCode=0 Nov 03 14:59:40.240654 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:40.240249 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" event={"ID":"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72","Type":"ContainerDied","Data":"66e6ed119d61905409ae68d61ad8d0061ab8d8b5e83ebe9931438ec7112e66c4"} Nov 03 14:59:40.330418 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:40.330384 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 14:59:41.393554 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.393529 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:41.477226 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.477192 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-openshift-ca\") pod \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " Nov 03 14:59:41.477419 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.477244 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-kube-api-access-fqdmj\" (UniqueName: \"kubernetes.io/projected/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-kube-api-access-fqdmj\") pod \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " Nov 03 14:59:41.477419 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.477274 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-xks-ca\") pod \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\" (UID: \"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72\") " Nov 03 14:59:41.477598 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.477576 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-openshift-ca" pod "c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72" (UID: "c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:59:41.477676 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.477659 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:59:41.479315 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.479294 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-xks-ca" pod "c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72" (UID: "c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:59:41.479409 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.479368 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-kube-api-access-fqdmj" pod "c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72" (UID: "c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72"). InnerVolumeSpecName "kube-api-access-fqdmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:59:41.579044 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.578964 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqdmj\" (UniqueName: \"kubernetes.io/projected/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-kube-api-access-fqdmj\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:59:41.579044 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:41.578993 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:59:42.253871 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:42.253838 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" event={"ID":"c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72","Type":"ContainerDied","Data":"de7dd48c7ebe79148d658db9a26d3bda8a424e7adb49583ae72a9cb32a549261"} Nov 03 14:59:42.254080 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:42.253879 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5071e830-qjkfg" Nov 03 14:59:42.254080 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:42.253887 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7dd48c7ebe79148d658db9a26d3bda8a424e7adb49583ae72a9cb32a549261" Nov 03 14:59:48.558102 ip-10-0-129-63 kubenswrapper[4924]: E1103 14:59:48.558049 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 14:59:49.326420 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.326385 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v"] Nov 03 14:59:49.326810 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.326797 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72" containerName="revoke-keys" Nov 03 14:59:49.326874 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.326811 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72" containerName="revoke-keys" Nov 03 14:59:49.326912 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.326897 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72" containerName="revoke-keys" Nov 03 14:59:49.330351 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.330330 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.379516 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.379483 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:59:49.425931 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.425892 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:59:49.428784 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.428755 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v"] Nov 03 14:59:49.450016 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.449979 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.450231 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.450033 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.450231 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.450145 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf22k\" (UniqueName: \"kubernetes.io/projected/dd72bad8-1540-4887-abaa-0684958d5186-kube-api-access-nf22k\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.551559 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.551514 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf22k\" (UniqueName: \"kubernetes.io/projected/dd72bad8-1540-4887-abaa-0684958d5186-kube-api-access-nf22k\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.551769 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.551593 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.551769 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.551646 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.552401 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.552375 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.553903 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.553884 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-xks-ca\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.606109 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.606020 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf22k\" (UniqueName: \"kubernetes.io/projected/dd72bad8-1540-4887-abaa-0684958d5186-kube-api-access-nf22k\") pod \"maas-api-revoke-sub-032958be-h5x6v\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:49.645670 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:49.645643 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:50.018423 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:50.018394 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v"] Nov 03 14:59:50.020993 ip-10-0-129-63 kubenswrapper[4924]: W1103 14:59:50.020961 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd72bad8_1540_4887_abaa_0684958d5186.slice/crio-7f14ea4f0dca8c4d4d2d90097cc309b252350ee5412ca0a261c4e80ca1e778f3 WatchSource:0}: Error finding container 7f14ea4f0dca8c4d4d2d90097cc309b252350ee5412ca0a261c4e80ca1e778f3: Status 404 returned error can't find the container with id 7f14ea4f0dca8c4d4d2d90097cc309b252350ee5412ca0a261c4e80ca1e778f3 Nov 03 14:59:50.299020 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:50.298929 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" event={"ID":"dd72bad8-1540-4887-abaa-0684958d5186","Type":"ContainerStarted","Data":"7f14ea4f0dca8c4d4d2d90097cc309b252350ee5412ca0a261c4e80ca1e778f3"} Nov 03 14:59:51.308515 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:51.308475 4924 generic.go:356] "Generic (PLEG): container finished" podID="dd72bad8-1540-4887-abaa-0684958d5186" containerID="9b6e33e086c0d0cb8cd7a3ae5bed141f8b0b723aaa38cfdbe079e709884c77ec" exitCode=0 Nov 03 14:59:51.308937 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:51.308565 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" event={"ID":"dd72bad8-1540-4887-abaa-0684958d5186","Type":"ContainerDied","Data":"9b6e33e086c0d0cb8cd7a3ae5bed141f8b0b723aaa38cfdbe079e709884c77ec"} Nov 03 14:59:52.453610 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.453580 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:52.580391 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.580321 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd72bad8-1540-4887-abaa-0684958d5186-kube-api-access-nf22k\" (UniqueName: \"kubernetes.io/projected/dd72bad8-1540-4887-abaa-0684958d5186-kube-api-access-nf22k\") pod \"dd72bad8-1540-4887-abaa-0684958d5186\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " Nov 03 14:59:52.580551 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.580404 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-xks-ca\") pod \"dd72bad8-1540-4887-abaa-0684958d5186\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " Nov 03 14:59:52.580551 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.580513 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-openshift-ca\") pod \"dd72bad8-1540-4887-abaa-0684958d5186\" (UID: \"dd72bad8-1540-4887-abaa-0684958d5186\") " Nov 03 14:59:52.580849 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.580826 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-openshift-ca" pod "dd72bad8-1540-4887-abaa-0684958d5186" (UID: "dd72bad8-1540-4887-abaa-0684958d5186"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:59:52.580953 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.580925 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:59:52.582437 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.582416 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-xks-ca" pod "dd72bad8-1540-4887-abaa-0684958d5186" (UID: "dd72bad8-1540-4887-abaa-0684958d5186"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:59:52.582562 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.582547 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd72bad8-1540-4887-abaa-0684958d5186-kube-api-access-nf22k" pod "dd72bad8-1540-4887-abaa-0684958d5186" (UID: "dd72bad8-1540-4887-abaa-0684958d5186"). InnerVolumeSpecName "kube-api-access-nf22k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:59:52.682340 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.682297 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf22k\" (UniqueName: \"kubernetes.io/projected/dd72bad8-1540-4887-abaa-0684958d5186-kube-api-access-nf22k\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:59:52.682340 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:52.682333 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd72bad8-1540-4887-abaa-0684958d5186-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:59:53.323178 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:53.323093 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" Nov 03 14:59:53.323343 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:53.323106 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-032958be-h5x6v" event={"ID":"dd72bad8-1540-4887-abaa-0684958d5186","Type":"ContainerDied","Data":"7f14ea4f0dca8c4d4d2d90097cc309b252350ee5412ca0a261c4e80ca1e778f3"} Nov 03 14:59:53.323343 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:53.323234 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f14ea4f0dca8c4d4d2d90097cc309b252350ee5412ca0a261c4e80ca1e778f3" Nov 03 14:59:55.313093 ip-10-0-129-63 kubenswrapper[4924]: I1103 14:59:55.312481 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 15:00:00.036452 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.036414 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm"] Nov 03 15:00:00.036934 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.036852 4924 cpu_manager.go:414] "containerMap: removing container" podUID="dd72bad8-1540-4887-abaa-0684958d5186" containerName="revoke-keys" Nov 03 15:00:00.036934 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.036866 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd72bad8-1540-4887-abaa-0684958d5186" containerName="revoke-keys" Nov 03 15:00:00.037104 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.036960 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd72bad8-1540-4887-abaa-0684958d5186" containerName="revoke-keys" Nov 03 15:00:00.040748 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.040726 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.090056 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.090016 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:00:00.136466 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.136433 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:00:00.139519 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.139496 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm"] Nov 03 15:00:00.163685 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.163651 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.163887 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.163709 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.163887 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.163818 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzm8\" (UniqueName: \"kubernetes.io/projected/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-kube-api-access-ltzm8\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.265210 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.265172 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.265417 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.265233 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.265417 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.265260 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzm8\" (UniqueName: \"kubernetes.io/projected/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-kube-api-access-ltzm8\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.265900 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.265875 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.267673 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.267650 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.319629 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.319559 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzm8\" (UniqueName: \"kubernetes.io/projected/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-kube-api-access-ltzm8\") pod \"maas-api-revoke-sub-3c79dfbb-gnwxm\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.354173 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.354134 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:00.723933 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:00.723905 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm"] Nov 03 15:00:00.725832 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:00:00.725800 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e426a6_4d2e_4275_a7d2_e7b7f23996b0.slice/crio-2173cf34a799bbc4c77404b105ebe5162058cc32ec77679826c9474174e3ed08 WatchSource:0}: Error finding container 2173cf34a799bbc4c77404b105ebe5162058cc32ec77679826c9474174e3ed08: Status 404 returned error can't find the container with id 2173cf34a799bbc4c77404b105ebe5162058cc32ec77679826c9474174e3ed08 Nov 03 15:00:01.394746 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:01.394658 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" event={"ID":"69e426a6-4d2e-4275-a7d2-e7b7f23996b0","Type":"ContainerStarted","Data":"5c478d22a1cc1f236ce3eba88b565462dc06509f0ee4ff41ec23a8334b3bb1a9"} Nov 03 15:00:01.394746 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:01.394719 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" event={"ID":"69e426a6-4d2e-4275-a7d2-e7b7f23996b0","Type":"ContainerStarted","Data":"2173cf34a799bbc4c77404b105ebe5162058cc32ec77679826c9474174e3ed08"} Nov 03 15:00:01.550480 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:01.550418 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" podStartSLOduration=2.550404953 podStartE2EDuration="2.550404953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:00:01.502638042 +0000 UTC m=+6934.697041637" watchObservedRunningTime="2026-11-03 15:00:01.550404953 +0000 UTC m=+6934.744808547" Nov 03 15:00:02.402044 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:02.402008 4924 generic.go:356] "Generic (PLEG): container finished" podID="69e426a6-4d2e-4275-a7d2-e7b7f23996b0" containerID="5c478d22a1cc1f236ce3eba88b565462dc06509f0ee4ff41ec23a8334b3bb1a9" exitCode=0 Nov 03 15:00:02.402490 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:02.402110 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" event={"ID":"69e426a6-4d2e-4275-a7d2-e7b7f23996b0","Type":"ContainerDied","Data":"5c478d22a1cc1f236ce3eba88b565462dc06509f0ee4ff41ec23a8334b3bb1a9"} Nov 03 15:00:03.547340 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.547315 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:03.595373 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.595336 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-kube-api-access-ltzm8\" (UniqueName: \"kubernetes.io/projected/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-kube-api-access-ltzm8\") pod \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " Nov 03 15:00:03.595558 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.595422 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-xks-ca\") pod \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " Nov 03 15:00:03.595558 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.595450 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-openshift-ca\") pod \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\" (UID: \"69e426a6-4d2e-4275-a7d2-e7b7f23996b0\") " Nov 03 15:00:03.595831 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.595805 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-openshift-ca" pod "69e426a6-4d2e-4275-a7d2-e7b7f23996b0" (UID: "69e426a6-4d2e-4275-a7d2-e7b7f23996b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:00:03.597447 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.597424 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-xks-ca" pod "69e426a6-4d2e-4275-a7d2-e7b7f23996b0" (UID: "69e426a6-4d2e-4275-a7d2-e7b7f23996b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:00:03.597542 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.597527 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-kube-api-access-ltzm8" pod "69e426a6-4d2e-4275-a7d2-e7b7f23996b0" (UID: "69e426a6-4d2e-4275-a7d2-e7b7f23996b0"). InnerVolumeSpecName "kube-api-access-ltzm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:00:03.697052 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.696956 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:03.697052 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.696985 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:03.697052 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:03.696995 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzm8\" (UniqueName: \"kubernetes.io/projected/69e426a6-4d2e-4275-a7d2-e7b7f23996b0-kube-api-access-ltzm8\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:04.414356 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:04.414325 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" Nov 03 15:00:04.414356 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:04.414338 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c79dfbb-gnwxm" event={"ID":"69e426a6-4d2e-4275-a7d2-e7b7f23996b0","Type":"ContainerDied","Data":"2173cf34a799bbc4c77404b105ebe5162058cc32ec77679826c9474174e3ed08"} Nov 03 15:00:04.414631 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:04.414381 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2173cf34a799bbc4c77404b105ebe5162058cc32ec77679826c9474174e3ed08" Nov 03 15:00:11.171358 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.171320 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml"] Nov 03 15:00:11.171954 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.171759 4924 cpu_manager.go:414] "containerMap: removing container" podUID="69e426a6-4d2e-4275-a7d2-e7b7f23996b0" containerName="revoke-keys" Nov 03 15:00:11.171954 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.171773 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e426a6-4d2e-4275-a7d2-e7b7f23996b0" containerName="revoke-keys" Nov 03 15:00:11.171954 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.171857 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e426a6-4d2e-4275-a7d2-e7b7f23996b0" containerName="revoke-keys" Nov 03 15:00:11.175381 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.175360 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.224373 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.224340 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:00:11.271118 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.271055 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:00:11.272577 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.272547 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.272682 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.272653 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.272682 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.272679 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5c2g\" (UniqueName: \"kubernetes.io/projected/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-kube-api-access-k5c2g\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.274057 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.274037 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml"] Nov 03 15:00:11.373382 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.373341 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.373605 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.373397 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5c2g\" (UniqueName: \"kubernetes.io/projected/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-kube-api-access-k5c2g\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.373605 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.373541 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.374175 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.374138 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.376042 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.376017 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.427745 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.427671 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5c2g\" (UniqueName: \"kubernetes.io/projected/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-kube-api-access-k5c2g\") pod \"maas-api-revoke-sub-fbf1c97d-pqmml\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.488153 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.488117 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:11.854623 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:11.854593 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml"] Nov 03 15:00:11.856117 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:00:11.856058 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c4ee06_a1fe_4a5e_a43c_d9f7cebeba66.slice/crio-69ffe04f0d8107dff407589beeb6903d1e38e8f94fe52cf51afded67754beea0 WatchSource:0}: Error finding container 69ffe04f0d8107dff407589beeb6903d1e38e8f94fe52cf51afded67754beea0: Status 404 returned error can't find the container with id 69ffe04f0d8107dff407589beeb6903d1e38e8f94fe52cf51afded67754beea0 Nov 03 15:00:12.471274 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:12.471204 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" event={"ID":"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66","Type":"ContainerStarted","Data":"e495c9df85844054853601259de970035d5c19321682aeaca79b02c19c254bca"} Nov 03 15:00:12.471751 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:12.471574 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" event={"ID":"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66","Type":"ContainerStarted","Data":"69ffe04f0d8107dff407589beeb6903d1e38e8f94fe52cf51afded67754beea0"} Nov 03 15:00:12.580444 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:12.580380 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" podStartSLOduration=1.5803662090000001 podStartE2EDuration="1.580366209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:00:12.578125182 +0000 UTC m=+6945.772528773" watchObservedRunningTime="2026-11-03 15:00:12.580366209 +0000 UTC m=+6945.774769802" Nov 03 15:00:13.480196 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:13.480154 4924 generic.go:356] "Generic (PLEG): container finished" podID="35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66" containerID="e495c9df85844054853601259de970035d5c19321682aeaca79b02c19c254bca" exitCode=0 Nov 03 15:00:13.480748 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:13.480240 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" event={"ID":"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66","Type":"ContainerDied","Data":"e495c9df85844054853601259de970035d5c19321682aeaca79b02c19c254bca"} Nov 03 15:00:14.840092 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:14.837171 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:14.920040 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:14.919900 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-openshift-ca\") pod \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " Nov 03 15:00:14.920040 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:14.919988 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-xks-ca\") pod \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " Nov 03 15:00:14.920296 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:14.920097 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-kube-api-access-k5c2g\" (UniqueName: \"kubernetes.io/projected/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-kube-api-access-k5c2g\") pod \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\" (UID: \"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66\") " Nov 03 15:00:14.920365 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:14.920342 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-openshift-ca" pod "35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66" (UID: "35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:00:14.920439 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:14.920421 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:14.922254 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:14.922222 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-xks-ca" pod "35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66" (UID: "35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:00:14.922382 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:14.922248 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-kube-api-access-k5c2g" pod "35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66" (UID: "35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66"). InnerVolumeSpecName "kube-api-access-k5c2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:00:15.021466 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:15.021406 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5c2g\" (UniqueName: \"kubernetes.io/projected/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-kube-api-access-k5c2g\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:15.021466 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:15.021440 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:15.512434 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:15.512378 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" event={"ID":"35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66","Type":"ContainerDied","Data":"69ffe04f0d8107dff407589beeb6903d1e38e8f94fe52cf51afded67754beea0"} Nov 03 15:00:15.512660 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:15.512452 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ffe04f0d8107dff407589beeb6903d1e38e8f94fe52cf51afded67754beea0" Nov 03 15:00:15.512660 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:15.512529 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf1c97d-pqmml" Nov 03 15:00:18.557581 ip-10-0-129-63 kubenswrapper[4924]: E1103 15:00:18.557543 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 15:00:19.119300 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:19.119265 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 15:00:22.372963 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.372926 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn"] Nov 03 15:00:22.373399 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.373372 4924 cpu_manager.go:414] "containerMap: removing container" podUID="35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66" containerName="revoke-keys" Nov 03 15:00:22.373399 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.373385 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66" containerName="revoke-keys" Nov 03 15:00:22.373491 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.373453 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66" containerName="revoke-keys" Nov 03 15:00:22.376969 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.376948 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.472409 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.472372 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:00:22.472660 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.472456 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:00:22.475538 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.475517 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn"] Nov 03 15:00:22.489288 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.489262 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.489413 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.489333 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gv99\" (UniqueName: \"kubernetes.io/projected/438e108e-fc46-4e29-898d-2b8caf89d60b-kube-api-access-2gv99\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.489511 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.489404 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.590008 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.589976 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gv99\" (UniqueName: \"kubernetes.io/projected/438e108e-fc46-4e29-898d-2b8caf89d60b-kube-api-access-2gv99\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.590008 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.590015 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.590270 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.590054 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.590705 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.590682 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.592504 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.592467 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.643722 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.643654 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gv99\" (UniqueName: \"kubernetes.io/projected/438e108e-fc46-4e29-898d-2b8caf89d60b-kube-api-access-2gv99\") pod \"maas-api-revoke-sub-a063136b-h2qtn\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:22.689769 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:22.689737 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:23.056086 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:23.056042 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn"] Nov 03 15:00:23.057012 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:00:23.056985 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438e108e_fc46_4e29_898d_2b8caf89d60b.slice/crio-945dedc0788e5ef5a61fcc2baf0782552f709dc18404c8a31e78d8ff712c077d WatchSource:0}: Error finding container 945dedc0788e5ef5a61fcc2baf0782552f709dc18404c8a31e78d8ff712c077d: Status 404 returned error can't find the container with id 945dedc0788e5ef5a61fcc2baf0782552f709dc18404c8a31e78d8ff712c077d Nov 03 15:00:23.560351 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:23.560307 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" event={"ID":"438e108e-fc46-4e29-898d-2b8caf89d60b","Type":"ContainerStarted","Data":"e3ea6ef20caa134173eaa711c243a33197425472edc0e39aabf4453c8752c767"} Nov 03 15:00:23.560767 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:23.560367 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" event={"ID":"438e108e-fc46-4e29-898d-2b8caf89d60b","Type":"ContainerStarted","Data":"945dedc0788e5ef5a61fcc2baf0782552f709dc18404c8a31e78d8ff712c077d"} Nov 03 15:00:23.673336 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:23.673281 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" podStartSLOduration=1.673266397 podStartE2EDuration="1.673266397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:00:23.671169002 +0000 UTC m=+6956.865572597" watchObservedRunningTime="2026-11-03 15:00:23.673266397 +0000 UTC m=+6956.867669990" Nov 03 15:00:24.567627 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:24.567588 4924 generic.go:356] "Generic (PLEG): container finished" podID="438e108e-fc46-4e29-898d-2b8caf89d60b" containerID="e3ea6ef20caa134173eaa711c243a33197425472edc0e39aabf4453c8752c767" exitCode=0 Nov 03 15:00:24.568032 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:24.567672 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" event={"ID":"438e108e-fc46-4e29-898d-2b8caf89d60b","Type":"ContainerDied","Data":"e3ea6ef20caa134173eaa711c243a33197425472edc0e39aabf4453c8752c767"} Nov 03 15:00:25.710837 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.710809 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:25.820178 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.820140 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-xks-ca\") pod \"438e108e-fc46-4e29-898d-2b8caf89d60b\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " Nov 03 15:00:25.820401 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.820194 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-openshift-ca\") pod \"438e108e-fc46-4e29-898d-2b8caf89d60b\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " Nov 03 15:00:25.820401 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.820322 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438e108e-fc46-4e29-898d-2b8caf89d60b-kube-api-access-2gv99\" (UniqueName: \"kubernetes.io/projected/438e108e-fc46-4e29-898d-2b8caf89d60b-kube-api-access-2gv99\") pod \"438e108e-fc46-4e29-898d-2b8caf89d60b\" (UID: \"438e108e-fc46-4e29-898d-2b8caf89d60b\") " Nov 03 15:00:25.820645 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.820617 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-openshift-ca" pod "438e108e-fc46-4e29-898d-2b8caf89d60b" (UID: "438e108e-fc46-4e29-898d-2b8caf89d60b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:00:25.822512 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.822477 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-xks-ca" pod "438e108e-fc46-4e29-898d-2b8caf89d60b" (UID: "438e108e-fc46-4e29-898d-2b8caf89d60b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:00:25.822644 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.822522 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438e108e-fc46-4e29-898d-2b8caf89d60b-kube-api-access-2gv99" pod "438e108e-fc46-4e29-898d-2b8caf89d60b" (UID: "438e108e-fc46-4e29-898d-2b8caf89d60b"). InnerVolumeSpecName "kube-api-access-2gv99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:00:25.922010 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.921977 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:25.922010 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.922010 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e108e-fc46-4e29-898d-2b8caf89d60b-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:25.922248 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:25.922027 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gv99\" (UniqueName: \"kubernetes.io/projected/438e108e-fc46-4e29-898d-2b8caf89d60b-kube-api-access-2gv99\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:26.586998 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:26.586952 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" event={"ID":"438e108e-fc46-4e29-898d-2b8caf89d60b","Type":"ContainerDied","Data":"945dedc0788e5ef5a61fcc2baf0782552f709dc18404c8a31e78d8ff712c077d"} Nov 03 15:00:26.587250 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:26.587023 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="945dedc0788e5ef5a61fcc2baf0782552f709dc18404c8a31e78d8ff712c077d" Nov 03 15:00:26.587250 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:26.587135 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a063136b-h2qtn" Nov 03 15:00:33.147385 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.147342 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w"] Nov 03 15:00:33.147987 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.147963 4924 cpu_manager.go:414] "containerMap: removing container" podUID="438e108e-fc46-4e29-898d-2b8caf89d60b" containerName="revoke-keys" Nov 03 15:00:33.148101 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.147989 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438e108e-fc46-4e29-898d-2b8caf89d60b" containerName="revoke-keys" Nov 03 15:00:33.148175 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.148113 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="438e108e-fc46-4e29-898d-2b8caf89d60b" containerName="revoke-keys" Nov 03 15:00:33.151665 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.151642 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.201144 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.201105 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:00:33.248356 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.248325 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:00:33.251350 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.251323 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w"] Nov 03 15:00:33.336346 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.336306 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.336523 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.336362 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g865s\" (UniqueName: \"kubernetes.io/projected/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-kube-api-access-g865s\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.336523 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.336383 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.437660 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.437620 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.437860 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.437683 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g865s\" (UniqueName: \"kubernetes.io/projected/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-kube-api-access-g865s\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.437860 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.437710 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.438456 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.438435 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.440146 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.440130 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.491401 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.491370 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g865s\" (UniqueName: \"kubernetes.io/projected/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-kube-api-access-g865s\") pod \"maas-api-revoke-sub-7c400fab-jl54w\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:33.764347 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:33.764265 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:34.132052 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:34.132022 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w"] Nov 03 15:00:34.132651 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:00:34.132616 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7264c16c_8466_4d05_bc17_07cd1c6e1f1c.slice/crio-e96c72aa39d076e64b3ee84493441c15c510076006414dbe65b693edb00f44c7 WatchSource:0}: Error finding container e96c72aa39d076e64b3ee84493441c15c510076006414dbe65b693edb00f44c7: Status 404 returned error can't find the container with id e96c72aa39d076e64b3ee84493441c15c510076006414dbe65b693edb00f44c7 Nov 03 15:00:34.642897 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:34.642852 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" event={"ID":"7264c16c-8466-4d05-bc17-07cd1c6e1f1c","Type":"ContainerStarted","Data":"0cab673a34d47a7f31a1cdd1b32bd9731cd61a446b41033f4cb13e34f567e84f"} Nov 03 15:00:34.643396 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:34.642913 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" event={"ID":"7264c16c-8466-4d05-bc17-07cd1c6e1f1c","Type":"ContainerStarted","Data":"e96c72aa39d076e64b3ee84493441c15c510076006414dbe65b693edb00f44c7"} Nov 03 15:00:34.753277 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:34.753184 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" podStartSLOduration=1.753170562 podStartE2EDuration="1.753170562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:00:34.749606597 +0000 UTC m=+6967.944010171" watchObservedRunningTime="2026-11-03 15:00:34.753170562 +0000 UTC m=+6967.947574156" Nov 03 15:00:35.650093 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:35.650029 4924 generic.go:356] "Generic (PLEG): container finished" podID="7264c16c-8466-4d05-bc17-07cd1c6e1f1c" containerID="0cab673a34d47a7f31a1cdd1b32bd9731cd61a446b41033f4cb13e34f567e84f" exitCode=0 Nov 03 15:00:35.650517 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:35.650103 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" event={"ID":"7264c16c-8466-4d05-bc17-07cd1c6e1f1c","Type":"ContainerDied","Data":"0cab673a34d47a7f31a1cdd1b32bd9731cd61a446b41033f4cb13e34f567e84f"} Nov 03 15:00:36.793815 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:36.793786 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:36.967635 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:36.967602 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-xks-ca\") pod \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " Nov 03 15:00:36.967782 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:36.967690 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-openshift-ca\") pod \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " Nov 03 15:00:36.967782 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:36.967770 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-kube-api-access-g865s\" (UniqueName: \"kubernetes.io/projected/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-kube-api-access-g865s\") pod \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\" (UID: \"7264c16c-8466-4d05-bc17-07cd1c6e1f1c\") " Nov 03 15:00:36.968037 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:36.968010 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-openshift-ca" pod "7264c16c-8466-4d05-bc17-07cd1c6e1f1c" (UID: "7264c16c-8466-4d05-bc17-07cd1c6e1f1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:00:36.969786 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:36.969765 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-xks-ca" pod "7264c16c-8466-4d05-bc17-07cd1c6e1f1c" (UID: "7264c16c-8466-4d05-bc17-07cd1c6e1f1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:00:36.969888 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:36.969780 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-kube-api-access-g865s" pod "7264c16c-8466-4d05-bc17-07cd1c6e1f1c" (UID: "7264c16c-8466-4d05-bc17-07cd1c6e1f1c"). InnerVolumeSpecName "kube-api-access-g865s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:00:37.069354 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:37.069320 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g865s\" (UniqueName: \"kubernetes.io/projected/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-kube-api-access-g865s\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:37.069354 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:37.069350 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:37.069354 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:37.069361 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7264c16c-8466-4d05-bc17-07cd1c6e1f1c-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:37.662644 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:37.662610 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" Nov 03 15:00:37.662644 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:37.662635 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c400fab-jl54w" event={"ID":"7264c16c-8466-4d05-bc17-07cd1c6e1f1c","Type":"ContainerDied","Data":"e96c72aa39d076e64b3ee84493441c15c510076006414dbe65b693edb00f44c7"} Nov 03 15:00:37.662864 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:37.662688 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e96c72aa39d076e64b3ee84493441c15c510076006414dbe65b693edb00f44c7" Nov 03 15:00:44.186223 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.186180 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw"] Nov 03 15:00:44.186895 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.186871 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7264c16c-8466-4d05-bc17-07cd1c6e1f1c" containerName="revoke-keys" Nov 03 15:00:44.186988 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.186897 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7264c16c-8466-4d05-bc17-07cd1c6e1f1c" containerName="revoke-keys" Nov 03 15:00:44.187062 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.187048 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7264c16c-8466-4d05-bc17-07cd1c6e1f1c" containerName="revoke-keys" Nov 03 15:00:44.190732 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.190711 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.226660 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.226627 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.226816 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.226697 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-454wz\" (UniqueName: \"kubernetes.io/projected/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-kube-api-access-454wz\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.226816 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.226790 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.239805 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.239744 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:00:44.239965 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.239910 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:00:44.248237 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.248208 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw"] Nov 03 15:00:44.327933 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.327895 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.328150 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.327963 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-454wz\" (UniqueName: \"kubernetes.io/projected/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-kube-api-access-454wz\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.328150 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.327985 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.328625 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.328599 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.330235 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.330216 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.383727 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.383695 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-454wz\" (UniqueName: \"kubernetes.io/projected/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-kube-api-access-454wz\") pod \"maas-api-revoke-sub-f08cc517-gjzpw\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.503226 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.503141 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:44.870852 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:44.870821 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw"] Nov 03 15:00:44.871534 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:00:44.871503 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode437d6fd_6758_437b_8db8_c3d06aa1d2b1.slice/crio-fa0bc48ab2f1aec897808b22484ae00286a03a7fca4b4e59966c86d28610734a WatchSource:0}: Error finding container fa0bc48ab2f1aec897808b22484ae00286a03a7fca4b4e59966c86d28610734a: Status 404 returned error can't find the container with id fa0bc48ab2f1aec897808b22484ae00286a03a7fca4b4e59966c86d28610734a Nov 03 15:00:45.712161 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:45.712116 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" event={"ID":"e437d6fd-6758-437b-8db8-c3d06aa1d2b1","Type":"ContainerStarted","Data":"9b6e23cd33b3f65859960297bb77afb4cc8d5be0635f5eee13e8be4556a5f09c"} Nov 03 15:00:45.712571 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:45.712183 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" event={"ID":"e437d6fd-6758-437b-8db8-c3d06aa1d2b1","Type":"ContainerStarted","Data":"fa0bc48ab2f1aec897808b22484ae00286a03a7fca4b4e59966c86d28610734a"} Nov 03 15:00:45.823444 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:45.823389 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" podStartSLOduration=1.823374756 podStartE2EDuration="1.823374756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:00:45.82109552 +0000 UTC m=+6979.015499115" watchObservedRunningTime="2026-11-03 15:00:45.823374756 +0000 UTC m=+6979.017778349" Nov 03 15:00:46.722156 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:46.722110 4924 generic.go:356] "Generic (PLEG): container finished" podID="e437d6fd-6758-437b-8db8-c3d06aa1d2b1" containerID="9b6e23cd33b3f65859960297bb77afb4cc8d5be0635f5eee13e8be4556a5f09c" exitCode=0 Nov 03 15:00:46.722716 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:46.722165 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" event={"ID":"e437d6fd-6758-437b-8db8-c3d06aa1d2b1","Type":"ContainerDied","Data":"9b6e23cd33b3f65859960297bb77afb4cc8d5be0635f5eee13e8be4556a5f09c"} Nov 03 15:00:48.059273 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.059236 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:48.121265 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.121231 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-xks-ca\") pod \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " Nov 03 15:00:48.121457 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.121335 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-openshift-ca\") pod \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " Nov 03 15:00:48.121457 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.121385 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-kube-api-access-454wz\" (UniqueName: \"kubernetes.io/projected/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-kube-api-access-454wz\") pod \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\" (UID: \"e437d6fd-6758-437b-8db8-c3d06aa1d2b1\") " Nov 03 15:00:48.122018 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.121994 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-openshift-ca" pod "e437d6fd-6758-437b-8db8-c3d06aa1d2b1" (UID: "e437d6fd-6758-437b-8db8-c3d06aa1d2b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:00:48.123495 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.123476 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-xks-ca" pod "e437d6fd-6758-437b-8db8-c3d06aa1d2b1" (UID: "e437d6fd-6758-437b-8db8-c3d06aa1d2b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:00:48.123601 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.123582 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-kube-api-access-454wz" pod "e437d6fd-6758-437b-8db8-c3d06aa1d2b1" (UID: "e437d6fd-6758-437b-8db8-c3d06aa1d2b1"). InnerVolumeSpecName "kube-api-access-454wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:00:48.222275 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.222240 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:48.222275 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.222269 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-454wz\" (UniqueName: \"kubernetes.io/projected/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-kube-api-access-454wz\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:48.222275 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.222280 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e437d6fd-6758-437b-8db8-c3d06aa1d2b1-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:48.626759 ip-10-0-129-63 kubenswrapper[4924]: E1103 15:00:48.626719 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 15:00:48.755095 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.754840 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" event={"ID":"e437d6fd-6758-437b-8db8-c3d06aa1d2b1","Type":"ContainerDied","Data":"fa0bc48ab2f1aec897808b22484ae00286a03a7fca4b4e59966c86d28610734a"} Nov 03 15:00:48.755095 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.754909 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa0bc48ab2f1aec897808b22484ae00286a03a7fca4b4e59966c86d28610734a" Nov 03 15:00:48.755095 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:48.754981 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cc517-gjzpw" Nov 03 15:00:55.133397 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.133352 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj"] Nov 03 15:00:55.133998 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.133972 4924 cpu_manager.go:414] "containerMap: removing container" podUID="e437d6fd-6758-437b-8db8-c3d06aa1d2b1" containerName="revoke-keys" Nov 03 15:00:55.134123 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.134000 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e437d6fd-6758-437b-8db8-c3d06aa1d2b1" containerName="revoke-keys" Nov 03 15:00:55.134197 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.134136 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e437d6fd-6758-437b-8db8-c3d06aa1d2b1" containerName="revoke-keys" Nov 03 15:00:55.137590 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.137566 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.187153 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.187120 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:00:55.202687 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.202652 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.202855 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.202716 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jp28\" (UniqueName: \"kubernetes.io/projected/c2312b8d-4aa0-4b98-b146-dc67c0dae766-kube-api-access-5jp28\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.202915 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.202851 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.233158 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.233125 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:00:55.236806 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.236779 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj"] Nov 03 15:00:55.304052 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.304006 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.304052 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.304050 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jp28\" (UniqueName: \"kubernetes.io/projected/c2312b8d-4aa0-4b98-b146-dc67c0dae766-kube-api-access-5jp28\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.304349 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.304101 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.304696 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.304672 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.306327 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.306293 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.359224 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.359187 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jp28\" (UniqueName: \"kubernetes.io/projected/c2312b8d-4aa0-4b98-b146-dc67c0dae766-kube-api-access-5jp28\") pod \"maas-api-revoke-sub-3de8f029-mb8fj\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.450963 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.450933 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:55.710230 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.710152 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 15:00:55.830140 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:55.830108 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj"] Nov 03 15:00:55.831203 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:00:55.831170 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2312b8d_4aa0_4b98_b146_dc67c0dae766.slice/crio-4ca8f7f9e885c5ae6d55f1067c1c181d81b51eb6903e37a9d067ede2d112b462 WatchSource:0}: Error finding container 4ca8f7f9e885c5ae6d55f1067c1c181d81b51eb6903e37a9d067ede2d112b462: Status 404 returned error can't find the container with id 4ca8f7f9e885c5ae6d55f1067c1c181d81b51eb6903e37a9d067ede2d112b462 Nov 03 15:00:56.802012 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:56.801974 4924 generic.go:356] "Generic (PLEG): container finished" podID="c2312b8d-4aa0-4b98-b146-dc67c0dae766" containerID="57632e3b76a161e88be023d3e119b5a62cf4b3f18e246e36651818795c85cefd" exitCode=0 Nov 03 15:00:56.802487 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:56.802066 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" event={"ID":"c2312b8d-4aa0-4b98-b146-dc67c0dae766","Type":"ContainerDied","Data":"57632e3b76a161e88be023d3e119b5a62cf4b3f18e246e36651818795c85cefd"} Nov 03 15:00:56.802487 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:56.802168 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" event={"ID":"c2312b8d-4aa0-4b98-b146-dc67c0dae766","Type":"ContainerStarted","Data":"4ca8f7f9e885c5ae6d55f1067c1c181d81b51eb6903e37a9d067ede2d112b462"} Nov 03 15:00:57.993801 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:57.993764 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:58.134021 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.133929 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-xks-ca\") pod \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " Nov 03 15:00:58.134243 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.134055 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2312b8d-4aa0-4b98-b146-dc67c0dae766-kube-api-access-5jp28\" (UniqueName: \"kubernetes.io/projected/c2312b8d-4aa0-4b98-b146-dc67c0dae766-kube-api-access-5jp28\") pod \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " Nov 03 15:00:58.134243 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.134123 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-openshift-ca\") pod \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\" (UID: \"c2312b8d-4aa0-4b98-b146-dc67c0dae766\") " Nov 03 15:00:58.134578 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.134547 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-openshift-ca" pod "c2312b8d-4aa0-4b98-b146-dc67c0dae766" (UID: "c2312b8d-4aa0-4b98-b146-dc67c0dae766"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:00:58.137690 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.137655 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-xks-ca" pod "c2312b8d-4aa0-4b98-b146-dc67c0dae766" (UID: "c2312b8d-4aa0-4b98-b146-dc67c0dae766"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:00:58.137827 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.137713 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2312b8d-4aa0-4b98-b146-dc67c0dae766-kube-api-access-5jp28" pod "c2312b8d-4aa0-4b98-b146-dc67c0dae766" (UID: "c2312b8d-4aa0-4b98-b146-dc67c0dae766"). InnerVolumeSpecName "kube-api-access-5jp28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:00:58.235454 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.235407 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jp28\" (UniqueName: \"kubernetes.io/projected/c2312b8d-4aa0-4b98-b146-dc67c0dae766-kube-api-access-5jp28\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:58.235454 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.235452 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:58.235761 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.235471 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2312b8d-4aa0-4b98-b146-dc67c0dae766-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:58.818701 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.818663 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" event={"ID":"c2312b8d-4aa0-4b98-b146-dc67c0dae766","Type":"ContainerDied","Data":"4ca8f7f9e885c5ae6d55f1067c1c181d81b51eb6903e37a9d067ede2d112b462"} Nov 03 15:00:58.818913 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.818721 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ca8f7f9e885c5ae6d55f1067c1c181d81b51eb6903e37a9d067ede2d112b462" Nov 03 15:00:58.818913 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:58.818787 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3de8f029-mb8fj" Nov 03 15:00:59.398569 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:59.398525 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:00:59.399212 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:59.399176 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dx6qt" podUID="43079ebb-3c46-4416-8822-bbe2be66119b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:00:59.546229 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:59.546189 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nmndv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:00:59.546514 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:00:59.546480 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nmndv" podUID="05cb7330-19b8-4112-b96b-cc70285298ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:01:06.032954 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.032916 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8"] Nov 03 15:01:06.033450 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.033431 4924 cpu_manager.go:414] "containerMap: removing container" podUID="c2312b8d-4aa0-4b98-b146-dc67c0dae766" containerName="revoke-keys" Nov 03 15:01:06.033503 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.033452 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2312b8d-4aa0-4b98-b146-dc67c0dae766" containerName="revoke-keys" Nov 03 15:01:06.033570 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.033560 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2312b8d-4aa0-4b98-b146-dc67c0dae766" containerName="revoke-keys" Nov 03 15:01:06.036923 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.036903 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.086102 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.086061 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:01:06.133001 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.132966 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:01:06.136198 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.136166 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8"] Nov 03 15:01:06.150866 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.150834 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.150992 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.150879 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.150992 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.150903 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x782\" (UniqueName: \"kubernetes.io/projected/6e0ab979-091d-4df8-8f19-39cf0e4b6417-kube-api-access-5x782\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.251926 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.251887 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.252143 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.251952 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.252143 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.251998 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x782\" (UniqueName: \"kubernetes.io/projected/6e0ab979-091d-4df8-8f19-39cf0e4b6417-kube-api-access-5x782\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.252633 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.252610 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.254564 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.254542 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.307950 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.307887 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x782\" (UniqueName: \"kubernetes.io/projected/6e0ab979-091d-4df8-8f19-39cf0e4b6417-kube-api-access-5x782\") pod \"maas-api-revoke-sub-62657ed0-459b8\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.350094 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.350049 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:06.716773 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.716747 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8"] Nov 03 15:01:06.717359 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:01:06.717327 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0ab979_091d_4df8_8f19_39cf0e4b6417.slice/crio-cd1390fbc99fbb07c49639e3ac05131917d85dc6b4740e7f78974b248e347456 WatchSource:0}: Error finding container cd1390fbc99fbb07c49639e3ac05131917d85dc6b4740e7f78974b248e347456: Status 404 returned error can't find the container with id cd1390fbc99fbb07c49639e3ac05131917d85dc6b4740e7f78974b248e347456 Nov 03 15:01:06.875941 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:06.875906 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" event={"ID":"6e0ab979-091d-4df8-8f19-39cf0e4b6417","Type":"ContainerStarted","Data":"cd1390fbc99fbb07c49639e3ac05131917d85dc6b4740e7f78974b248e347456"} Nov 03 15:01:07.883738 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:07.883645 4924 generic.go:356] "Generic (PLEG): container finished" podID="6e0ab979-091d-4df8-8f19-39cf0e4b6417" containerID="bcabe38b04daab8506d2dc9f2f445fd22508242ef80c3623935dcb8fbbe6e749" exitCode=0 Nov 03 15:01:07.883738 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:07.883721 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" event={"ID":"6e0ab979-091d-4df8-8f19-39cf0e4b6417","Type":"ContainerDied","Data":"bcabe38b04daab8506d2dc9f2f445fd22508242ef80c3623935dcb8fbbe6e749"} Nov 03 15:01:09.057476 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.057443 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:09.077498 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.077463 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-xks-ca\") pod \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " Nov 03 15:01:09.077676 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.077536 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-openshift-ca\") pod \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " Nov 03 15:01:09.077676 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.077636 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0ab979-091d-4df8-8f19-39cf0e4b6417-kube-api-access-5x782\" (UniqueName: \"kubernetes.io/projected/6e0ab979-091d-4df8-8f19-39cf0e4b6417-kube-api-access-5x782\") pod \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\" (UID: \"6e0ab979-091d-4df8-8f19-39cf0e4b6417\") " Nov 03 15:01:09.077933 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.077907 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-openshift-ca" pod "6e0ab979-091d-4df8-8f19-39cf0e4b6417" (UID: "6e0ab979-091d-4df8-8f19-39cf0e4b6417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:01:09.079810 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.079776 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0ab979-091d-4df8-8f19-39cf0e4b6417-kube-api-access-5x782" pod "6e0ab979-091d-4df8-8f19-39cf0e4b6417" (UID: "6e0ab979-091d-4df8-8f19-39cf0e4b6417"). InnerVolumeSpecName "kube-api-access-5x782". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:01:09.080059 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.080034 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-xks-ca" pod "6e0ab979-091d-4df8-8f19-39cf0e4b6417" (UID: "6e0ab979-091d-4df8-8f19-39cf0e4b6417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:01:09.178949 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.178914 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:09.178949 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.178944 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0ab979-091d-4df8-8f19-39cf0e4b6417-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:09.178949 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.178954 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x782\" (UniqueName: \"kubernetes.io/projected/6e0ab979-091d-4df8-8f19-39cf0e4b6417-kube-api-access-5x782\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:09.905054 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.905018 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" Nov 03 15:01:09.905405 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.905021 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62657ed0-459b8" event={"ID":"6e0ab979-091d-4df8-8f19-39cf0e4b6417","Type":"ContainerDied","Data":"cd1390fbc99fbb07c49639e3ac05131917d85dc6b4740e7f78974b248e347456"} Nov 03 15:01:09.905532 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:09.905458 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd1390fbc99fbb07c49639e3ac05131917d85dc6b4740e7f78974b248e347456" Nov 03 15:01:10.904743 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:10.898248 4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-sw482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:01:10.904743 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:10.898342 4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-sw482" podUID="f586dcb0-04d2-4152-9556-931824aab29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:01:11.048049 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:11.048010 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt"] Nov 03 15:01:11.059685 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:11.059652 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f6b482d-dsrwt"] Nov 03 15:01:12.949862 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:12.949825 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b526644-eb84-4cae-82bd-350ea2bc6dcc" path="/var/lib/kubelet/pods/2b526644-eb84-4cae-82bd-350ea2bc6dcc/volumes" Nov 03 15:01:16.844673 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.844633 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl"] Nov 03 15:01:16.845316 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.845300 4924 cpu_manager.go:414] "containerMap: removing container" podUID="6e0ab979-091d-4df8-8f19-39cf0e4b6417" containerName="revoke-keys" Nov 03 15:01:16.845402 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.845323 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0ab979-091d-4df8-8f19-39cf0e4b6417" containerName="revoke-keys" Nov 03 15:01:16.845477 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.845461 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0ab979-091d-4df8-8f19-39cf0e4b6417" containerName="revoke-keys" Nov 03 15:01:16.850196 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.850174 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:16.901504 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.901470 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:01:16.907269 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.907227 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:16.907416 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.907348 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qvc\" (UniqueName: \"kubernetes.io/projected/7e88c89b-3863-459a-97d3-2ad058e93561-kube-api-access-z5qvc\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:16.907416 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.907385 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:16.948381 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.948344 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:01:16.952235 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:16.952208 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl"] Nov 03 15:01:17.008235 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.008200 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qvc\" (UniqueName: \"kubernetes.io/projected/7e88c89b-3863-459a-97d3-2ad058e93561-kube-api-access-z5qvc\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:17.008474 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.008255 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:17.008474 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.008413 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:17.009314 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.009281 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:17.010847 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.010821 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:17.062815 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.062787 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qvc\" (UniqueName: \"kubernetes.io/projected/7e88c89b-3863-459a-97d3-2ad058e93561-kube-api-access-z5qvc\") pod \"maas-api-revoke-sub-71534dcf-hsggl\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:17.163686 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.163640 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:17.538001 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.537976 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl"] Nov 03 15:01:17.539747 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:01:17.539714 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e88c89b_3863_459a_97d3_2ad058e93561.slice/crio-93ec2aefc4f30710f49e489a6bdbdcd3f969cd1fc64758c739c8908e705dec34 WatchSource:0}: Error finding container 93ec2aefc4f30710f49e489a6bdbdcd3f969cd1fc64758c739c8908e705dec34: Status 404 returned error can't find the container with id 93ec2aefc4f30710f49e489a6bdbdcd3f969cd1fc64758c739c8908e705dec34 Nov 03 15:01:17.965167 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.965123 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" event={"ID":"7e88c89b-3863-459a-97d3-2ad058e93561","Type":"ContainerStarted","Data":"38717c9fc42af062faa3582cdc90c3529134a3df890b62fe227317f167d6b55d"} Nov 03 15:01:17.965543 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:17.965216 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" event={"ID":"7e88c89b-3863-459a-97d3-2ad058e93561","Type":"ContainerStarted","Data":"93ec2aefc4f30710f49e489a6bdbdcd3f969cd1fc64758c739c8908e705dec34"} Nov 03 15:01:18.076806 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:18.076746 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" podStartSLOduration=2.076730706 podStartE2EDuration="2.076730706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:01:18.074212475 +0000 UTC m=+7011.268616069" watchObservedRunningTime="2026-11-03 15:01:18.076730706 +0000 UTC m=+7011.271134300" Nov 03 15:01:18.558591 ip-10-0-129-63 kubenswrapper[4924]: E1103 15:01:18.558547 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 15:01:18.972931 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:18.972894 4924 generic.go:356] "Generic (PLEG): container finished" podID="7e88c89b-3863-459a-97d3-2ad058e93561" containerID="38717c9fc42af062faa3582cdc90c3529134a3df890b62fe227317f167d6b55d" exitCode=0 Nov 03 15:01:18.973372 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:18.972975 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" event={"ID":"7e88c89b-3863-459a-97d3-2ad058e93561","Type":"ContainerDied","Data":"38717c9fc42af062faa3582cdc90c3529134a3df890b62fe227317f167d6b55d"} Nov 03 15:01:20.190095 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.181040 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:20.256310 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.255336 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-xks-ca\") pod \"7e88c89b-3863-459a-97d3-2ad058e93561\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " Nov 03 15:01:20.256310 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.255446 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-openshift-ca\") pod \"7e88c89b-3863-459a-97d3-2ad058e93561\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " Nov 03 15:01:20.256310 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.255483 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e88c89b-3863-459a-97d3-2ad058e93561-kube-api-access-z5qvc\" (UniqueName: \"kubernetes.io/projected/7e88c89b-3863-459a-97d3-2ad058e93561-kube-api-access-z5qvc\") pod \"7e88c89b-3863-459a-97d3-2ad058e93561\" (UID: \"7e88c89b-3863-459a-97d3-2ad058e93561\") " Nov 03 15:01:20.260343 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.259418 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-openshift-ca" pod "7e88c89b-3863-459a-97d3-2ad058e93561" (UID: "7e88c89b-3863-459a-97d3-2ad058e93561"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:01:20.271560 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.271302 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-xks-ca" pod "7e88c89b-3863-459a-97d3-2ad058e93561" (UID: "7e88c89b-3863-459a-97d3-2ad058e93561"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:01:20.290714 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.290449 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e88c89b-3863-459a-97d3-2ad058e93561-kube-api-access-z5qvc" pod "7e88c89b-3863-459a-97d3-2ad058e93561" (UID: "7e88c89b-3863-459a-97d3-2ad058e93561"). InnerVolumeSpecName "kube-api-access-z5qvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:01:20.356911 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.356861 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:20.357230 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.357204 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e88c89b-3863-459a-97d3-2ad058e93561-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:20.367642 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:20.367615 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qvc\" (UniqueName: \"kubernetes.io/projected/7e88c89b-3863-459a-97d3-2ad058e93561-kube-api-access-z5qvc\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:21.001036 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:21.000997 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" event={"ID":"7e88c89b-3863-459a-97d3-2ad058e93561","Type":"ContainerDied","Data":"93ec2aefc4f30710f49e489a6bdbdcd3f969cd1fc64758c739c8908e705dec34"} Nov 03 15:01:21.001389 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:21.001368 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ec2aefc4f30710f49e489a6bdbdcd3f969cd1fc64758c739c8908e705dec34" Nov 03 15:01:21.001589 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:21.001575 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71534dcf-hsggl" Nov 03 15:01:22.048718 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:22.048681 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr"] Nov 03 15:01:22.095466 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:22.095414 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4df84cea-dktzr"] Nov 03 15:01:22.949690 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:22.949653 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f11bfb-85a4-44b6-8018-a3c8b0cde046" path="/var/lib/kubelet/pods/a7f11bfb-85a4-44b6-8018-a3c8b0cde046/volumes" Nov 03 15:01:27.621142 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.621102 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f"] Nov 03 15:01:27.621566 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.621551 4924 cpu_manager.go:414] "containerMap: removing container" podUID="7e88c89b-3863-459a-97d3-2ad058e93561" containerName="revoke-keys" Nov 03 15:01:27.621620 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.621568 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e88c89b-3863-459a-97d3-2ad058e93561" containerName="revoke-keys" Nov 03 15:01:27.621654 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.621635 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e88c89b-3863-459a-97d3-2ad058e93561" containerName="revoke-keys" Nov 03 15:01:27.625208 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.625189 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.674804 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.674772 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:01:27.721223 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.721186 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:01:27.724561 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.724536 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f"] Nov 03 15:01:27.763776 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.763737 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xrkf\" (UniqueName: \"kubernetes.io/projected/3625138c-53b7-4454-b180-4af090c9bdfd-kube-api-access-8xrkf\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.763945 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.763878 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.763945 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.763930 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.864508 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.864476 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrkf\" (UniqueName: \"kubernetes.io/projected/3625138c-53b7-4454-b180-4af090c9bdfd-kube-api-access-8xrkf\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.864691 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.864592 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.864691 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.864619 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.865311 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.865287 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.866876 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.866859 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.918508 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.918472 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrkf\" (UniqueName: \"kubernetes.io/projected/3625138c-53b7-4454-b180-4af090c9bdfd-kube-api-access-8xrkf\") pod \"maas-api-revoke-sub-49589005-k277f\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:27.939310 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:27.939282 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:28.316789 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:01:28.316753 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3625138c_53b7_4454_b180_4af090c9bdfd.slice/crio-7cb99232211f95f4cea5f2908efcbae5a843eb3a14d52c6c5f0b1312881be2f3 WatchSource:0}: Error finding container 7cb99232211f95f4cea5f2908efcbae5a843eb3a14d52c6c5f0b1312881be2f3: Status 404 returned error can't find the container with id 7cb99232211f95f4cea5f2908efcbae5a843eb3a14d52c6c5f0b1312881be2f3 Nov 03 15:01:28.316978 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:28.316938 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f"] Nov 03 15:01:29.054419 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:29.054382 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" event={"ID":"3625138c-53b7-4454-b180-4af090c9bdfd","Type":"ContainerStarted","Data":"d605d975e367f7a02092ff54c58f7d552e8488cd3eca4c0cda38d9b45ec5d20d"} Nov 03 15:01:29.054837 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:29.054449 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" event={"ID":"3625138c-53b7-4454-b180-4af090c9bdfd","Type":"ContainerStarted","Data":"7cb99232211f95f4cea5f2908efcbae5a843eb3a14d52c6c5f0b1312881be2f3"} Nov 03 15:01:29.162899 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:29.162848 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" podStartSLOduration=2.162829011 podStartE2EDuration="2.162829011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:01:29.161926175 +0000 UTC m=+7022.356329787" watchObservedRunningTime="2026-11-03 15:01:29.162829011 +0000 UTC m=+7022.357232605" Nov 03 15:01:30.061982 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:30.061933 4924 generic.go:356] "Generic (PLEG): container finished" podID="3625138c-53b7-4454-b180-4af090c9bdfd" containerID="d605d975e367f7a02092ff54c58f7d552e8488cd3eca4c0cda38d9b45ec5d20d" exitCode=0 Nov 03 15:01:30.062418 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:30.062052 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" event={"ID":"3625138c-53b7-4454-b180-4af090c9bdfd","Type":"ContainerDied","Data":"d605d975e367f7a02092ff54c58f7d552e8488cd3eca4c0cda38d9b45ec5d20d"} Nov 03 15:01:31.612592 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.612562 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:31.694103 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.693620 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3625138c-53b7-4454-b180-4af090c9bdfd-kube-api-access-8xrkf\" (UniqueName: \"kubernetes.io/projected/3625138c-53b7-4454-b180-4af090c9bdfd-kube-api-access-8xrkf\") pod \"3625138c-53b7-4454-b180-4af090c9bdfd\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " Nov 03 15:01:31.694378 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.694355 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-xks-ca\") pod \"3625138c-53b7-4454-b180-4af090c9bdfd\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " Nov 03 15:01:31.694638 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.694618 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-openshift-ca\") pod \"3625138c-53b7-4454-b180-4af090c9bdfd\" (UID: \"3625138c-53b7-4454-b180-4af090c9bdfd\") " Nov 03 15:01:31.702230 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.702194 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3625138c-53b7-4454-b180-4af090c9bdfd-kube-api-access-8xrkf" pod "3625138c-53b7-4454-b180-4af090c9bdfd" (UID: "3625138c-53b7-4454-b180-4af090c9bdfd"). InnerVolumeSpecName "kube-api-access-8xrkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:01:31.702515 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.702491 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-openshift-ca" pod "3625138c-53b7-4454-b180-4af090c9bdfd" (UID: "3625138c-53b7-4454-b180-4af090c9bdfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:01:31.702789 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.702760 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-xks-ca" pod "3625138c-53b7-4454-b180-4af090c9bdfd" (UID: "3625138c-53b7-4454-b180-4af090c9bdfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:01:31.797829 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.797779 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:31.798135 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.798095 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xrkf\" (UniqueName: \"kubernetes.io/projected/3625138c-53b7-4454-b180-4af090c9bdfd-kube-api-access-8xrkf\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:31.798287 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:31.798262 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3625138c-53b7-4454-b180-4af090c9bdfd-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:32.105490 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:32.105439 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" event={"ID":"3625138c-53b7-4454-b180-4af090c9bdfd","Type":"ContainerDied","Data":"7cb99232211f95f4cea5f2908efcbae5a843eb3a14d52c6c5f0b1312881be2f3"} Nov 03 15:01:32.105722 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:32.105517 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cb99232211f95f4cea5f2908efcbae5a843eb3a14d52c6c5f0b1312881be2f3" Nov 03 15:01:32.105722 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:32.105588 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49589005-k277f" Nov 03 15:01:38.382520 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.382388 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8"] Nov 03 15:01:38.383249 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.383218 4924 cpu_manager.go:414] "containerMap: removing container" podUID="3625138c-53b7-4454-b180-4af090c9bdfd" containerName="revoke-keys" Nov 03 15:01:38.383249 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.383247 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3625138c-53b7-4454-b180-4af090c9bdfd" containerName="revoke-keys" Nov 03 15:01:38.383448 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.383387 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3625138c-53b7-4454-b180-4af090c9bdfd" containerName="revoke-keys" Nov 03 15:01:38.389783 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.389757 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.441397 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.441364 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 15:01:38.452129 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.452097 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.452271 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.452140 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.452333 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.452275 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbbp\" (UniqueName: \"kubernetes.io/projected/b1dc3e58-86d3-4386-8204-22ba0f147053-kube-api-access-zgbbp\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.488046 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.488013 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:01:38.491290 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.491261 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8"] Nov 03 15:01:38.552844 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.552814 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbbp\" (UniqueName: \"kubernetes.io/projected/b1dc3e58-86d3-4386-8204-22ba0f147053-kube-api-access-zgbbp\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.553005 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.552851 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.553005 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.552880 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.553581 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.553556 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.555291 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.555270 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.607749 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.607714 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbbp\" (UniqueName: \"kubernetes.io/projected/b1dc3e58-86d3-4386-8204-22ba0f147053-kube-api-access-zgbbp\") pod \"maas-api-revoke-sub-0dbd9187-dr8v8\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:38.703745 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:38.703711 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:39.073341 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:39.073311 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8"] Nov 03 15:01:39.075580 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:01:39.075549 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1dc3e58_86d3_4386_8204_22ba0f147053.slice/crio-c0f4644c696d81e70988b51fb64d4a07034fac4cc17a9646f175a033db31213a WatchSource:0}: Error finding container c0f4644c696d81e70988b51fb64d4a07034fac4cc17a9646f175a033db31213a: Status 404 returned error can't find the container with id c0f4644c696d81e70988b51fb64d4a07034fac4cc17a9646f175a033db31213a Nov 03 15:01:39.147619 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:39.147585 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" event={"ID":"b1dc3e58-86d3-4386-8204-22ba0f147053","Type":"ContainerStarted","Data":"c0f4644c696d81e70988b51fb64d4a07034fac4cc17a9646f175a033db31213a"} Nov 03 15:01:40.154843 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:40.154751 4924 generic.go:356] "Generic (PLEG): container finished" podID="b1dc3e58-86d3-4386-8204-22ba0f147053" containerID="52ff61b9c0b39279d2c3ba06a9bb63a832b2b84a186b1cdc4321d0c68ec6319b" exitCode=0 Nov 03 15:01:40.155239 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:40.154840 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" event={"ID":"b1dc3e58-86d3-4386-8204-22ba0f147053","Type":"ContainerDied","Data":"52ff61b9c0b39279d2c3ba06a9bb63a832b2b84a186b1cdc4321d0c68ec6319b"} Nov 03 15:01:41.540578 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.540255 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:41.608532 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.608480 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-xks-ca\") pod \"b1dc3e58-86d3-4386-8204-22ba0f147053\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " Nov 03 15:01:41.608717 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.608537 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1dc3e58-86d3-4386-8204-22ba0f147053-kube-api-access-zgbbp\" (UniqueName: \"kubernetes.io/projected/b1dc3e58-86d3-4386-8204-22ba0f147053-kube-api-access-zgbbp\") pod \"b1dc3e58-86d3-4386-8204-22ba0f147053\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " Nov 03 15:01:41.613098 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.612443 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1dc3e58-86d3-4386-8204-22ba0f147053-kube-api-access-zgbbp" pod "b1dc3e58-86d3-4386-8204-22ba0f147053" (UID: "b1dc3e58-86d3-4386-8204-22ba0f147053"). InnerVolumeSpecName "kube-api-access-zgbbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:01:41.613098 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.612498 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-xks-ca" pod "b1dc3e58-86d3-4386-8204-22ba0f147053" (UID: "b1dc3e58-86d3-4386-8204-22ba0f147053"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 15:01:41.710119 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.709433 4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-openshift-ca\") pod \"b1dc3e58-86d3-4386-8204-22ba0f147053\" (UID: \"b1dc3e58-86d3-4386-8204-22ba0f147053\") " Nov 03 15:01:41.710119 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.709807 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-xks-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:41.710119 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.709832 4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbbp\" (UniqueName: \"kubernetes.io/projected/b1dc3e58-86d3-4386-8204-22ba0f147053-kube-api-access-zgbbp\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:41.710665 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.710619 4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-openshift-ca" pod "b1dc3e58-86d3-4386-8204-22ba0f147053" (UID: "b1dc3e58-86d3-4386-8204-22ba0f147053"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:01:41.812698 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:41.812601 4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1dc3e58-86d3-4386-8204-22ba0f147053-maas-api-openshift-ca\") on node \"ip-10-0-129-63.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:01:42.191803 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:42.191761 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" event={"ID":"b1dc3e58-86d3-4386-8204-22ba0f147053","Type":"ContainerDied","Data":"c0f4644c696d81e70988b51fb64d4a07034fac4cc17a9646f175a033db31213a"} Nov 03 15:01:42.192062 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:42.192043 4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f4644c696d81e70988b51fb64d4a07034fac4cc17a9646f175a033db31213a" Nov 03 15:01:42.192302 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:42.192284 4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbd9187-dr8v8" Nov 03 15:01:48.557960 ip-10-0-129-63 kubenswrapper[4924]: E1103 15:01:48.557916 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 15:01:56.062337 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:56.062280 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 15:01:57.048035 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:57.047836 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h"] Nov 03 15:01:57.091872 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:57.091832 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe9d177-4qb5h"] Nov 03 15:01:58.950208 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:01:58.950175 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e82ee7a-f7a5-4af9-86a0-266c30de9330" path="/var/lib/kubelet/pods/2e82ee7a-f7a5-4af9-86a0-266c30de9330/volumes" Nov 03 15:02:18.558112 ip-10-0-129-63 kubenswrapper[4924]: E1103 15:02:18.558001 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 15:02:20.206680 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:20.206646 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwnhg_19a01435-9815-42ed-9e2d-324a54f125bd/manager/0.log" Nov 03 15:02:20.436180 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:20.436143 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8kq74_dfd95a5f-d506-4461-8baf-44d26ec69181/manager/0.log" Nov 03 15:02:20.707215 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:20.707179 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xbk2m_bde28c41-ee98-42d6-ad61-6d08466d2fc9/manager/0.log" Nov 03 15:02:24.629794 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:24.629769 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-p9scb_d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb/server/0.log" Nov 03 15:02:24.849894 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:24.849857 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pghpt_3ff5388d-d285-446d-9ccc-98a7be123067/manager/0.log" Nov 03 15:02:27.039466 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:27.039435 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-032958be-h5x6v_dd72bad8-1540-4887-abaa-0684958d5186/revoke-keys/0.log" Nov 03 15:02:27.263991 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:27.263958 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0dbd9187-dr8v8_b1dc3e58-86d3-4386-8204-22ba0f147053/revoke-keys/0.log" Nov 03 15:02:27.484096 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:27.484040 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3c79dfbb-gnwxm_69e426a6-4d2e-4275-a7d2-e7b7f23996b0/revoke-keys/0.log" Nov 03 15:02:27.700976 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:27.700945 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3de8f029-mb8fj_c2312b8d-4aa0-4b98-b146-dc67c0dae766/revoke-keys/0.log" Nov 03 15:02:27.914300 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:27.914267 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4155d965-vpwm2_e12ea4bc-a26d-4e82-963a-767e53a3e27f/revoke-keys/0.log" Nov 03 15:02:28.131739 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:28.131711 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-49589005-k277f_3625138c-53b7-4454-b180-4af090c9bdfd/revoke-keys/0.log" Nov 03 15:02:28.363529 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:28.363453 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5071e830-qjkfg_c4782f6b-4f8e-40f7-a4ae-d9f84fed5c72/revoke-keys/0.log" Nov 03 15:02:28.576682 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:28.576650 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-62657ed0-459b8_6e0ab979-091d-4df8-8f19-39cf0e4b6417/revoke-keys/0.log" Nov 03 15:02:28.806645 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:28.806613 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-71534dcf-hsggl_7e88c89b-3863-459a-97d3-2ad058e93561/revoke-keys/0.log" Nov 03 15:02:29.023970 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:29.023943 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7c400fab-jl54w_7264c16c-8466-4d05-bc17-07cd1c6e1f1c/revoke-keys/0.log" Nov 03 15:02:29.133802 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:29.133718 4924 scope.go:122] "RemoveContainer" containerID="b11adaffa08931d8da3b8caefcb73042076703ff53aa7e3f3d30f3487e37006b" Nov 03 15:02:29.143316 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:29.143294 4924 scope.go:122] "RemoveContainer" containerID="ea4aa85a5c4973a72b7972e0baf2fb52f50adec6682c3bcab982ab8c986e2f73" Nov 03 15:02:29.247171 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:29.247144 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8c40aa77-bld6t_a32c63dd-a70d-43ee-a5a0-126c2c84b33d/revoke-keys/0.log" Nov 03 15:02:29.470434 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:29.470405 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a063136b-h2qtn_438e108e-fc46-4e29-898d-2b8caf89d60b/revoke-keys/0.log" Nov 03 15:02:29.686089 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:29.686046 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aeb9e5aa-qn4m9_e79d57cc-9fac-4ee6-9c2c-a07cad9060cc/revoke-keys/0.log" Nov 03 15:02:29.902704 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:29.902634 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d4e2751f-jm7hp_354b0c41-d41a-4fc6-83c4-9b7beaf3659d/revoke-keys/0.log" Nov 03 15:02:30.123321 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:30.123293 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f08cc517-gjzpw_e437d6fd-6758-437b-8db8-c3d06aa1d2b1/revoke-keys/0.log" Nov 03 15:02:30.351148 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:30.351116 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fbf1c97d-pqmml_35c4ee06-a1fe-4a5e-a43c-d9f7cebeba66/revoke-keys/0.log" Nov 03 15:02:30.570863 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:30.570832 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jpdjs_4a2dd214-4f0c-4299-aa46-d35a67280faf/postgres/0.log" Nov 03 15:02:32.977713 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:32.977678 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qkwsd_1d3c8424-3ae1-488e-b37f-1d2a5d6c5199/manager/0.log" Nov 03 15:02:33.213025 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:33.212994 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fvdmt_199f6b24-0d49-4637-b048-17b035f48627/manager/0.log" Nov 03 15:02:33.681516 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:33.681484 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-x2vzl_a444f2ee-fc8a-4780-8058-3e5e6c2fae73/manager/0.log" Nov 03 15:02:34.284952 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:34.284919 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mvjp9_7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a/manager/0.log" Nov 03 15:02:35.327087 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:35.327036 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fc46679f7cttvl_2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8/istio-proxy/0.log" Nov 03 15:02:37.187888 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:37.187857 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-dx6qt_43079ebb-3c46-4416-8822-bbe2be66119b/istio-proxy/0.log" Nov 03 15:02:38.589898 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:38.589868 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-nmndv_05cb7330-19b8-4112-b96b-cc70285298ed/istio-proxy/0.log" Nov 03 15:02:38.821754 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:38.821725 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-sw482_f586dcb0-04d2-4152-9556-931824aab29b/istio-proxy/0.log" Nov 03 15:02:39.052336 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:39.052303 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95_0db0b8cf-03c4-4493-a3f8-464723ef7eec/istio-proxy/0.log" Nov 03 15:02:39.717899 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:39.717866 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8478467d49-l5dtm_f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/router/0.log" Nov 03 15:02:41.047380 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:41.047339 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t"] Nov 03 15:02:41.094144 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:41.094116 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c40aa77-bld6t"] Nov 03 15:02:42.949613 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:42.949581 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32c63dd-a70d-43ee-a5a0-126c2c84b33d" path="/var/lib/kubelet/pods/a32c63dd-a70d-43ee-a5a0-126c2c84b33d/volumes" Nov 03 15:02:48.558231 ip-10-0-129-63 kubenswrapper[4924]: E1103 15:02:48.558192 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 15:02:52.047658 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:52.047623 4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9"] Nov 03 15:02:52.093816 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:52.093787 4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb9e5aa-qn4m9"] Nov 03 15:02:52.957921 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:52.957886 4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79d57cc-9fac-4ee6-9c2c-a07cad9060cc" path="/var/lib/kubelet/pods/e79d57cc-9fac-4ee6-9c2c-a07cad9060cc/volumes" Nov 03 15:02:53.786763 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.786725 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 15:02:53.851373 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.851335 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wn4x5/must-gather-hbbcr"] Nov 03 15:02:53.851738 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.851725 4924 cpu_manager.go:414] "containerMap: removing container" podUID="b1dc3e58-86d3-4386-8204-22ba0f147053" containerName="revoke-keys" Nov 03 15:02:53.851787 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.851740 4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1dc3e58-86d3-4386-8204-22ba0f147053" containerName="revoke-keys" Nov 03 15:02:53.851846 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.851837 4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1dc3e58-86d3-4386-8204-22ba0f147053" containerName="revoke-keys" Nov 03 15:02:53.855920 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.855898 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wn4x5/must-gather-hbbcr" Nov 03 15:02:53.934632 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.934597 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 15:02:53.953734 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.953706 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wn4x5\"/\"kube-root-ca.crt\"" Nov 03 15:02:53.954012 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.953997 4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wn4x5\"/\"openshift-service-ca.crt\"" Nov 03 15:02:53.954102 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.954027 4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wn4x5\"/\"default-dockercfg-gmmrj\"" Nov 03 15:02:53.956985 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.956966 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wn4x5/must-gather-hbbcr"] Nov 03 15:02:53.986625 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.986586 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92818b3a-57ba-43fa-823c-01d3bbdffef1-must-gather-output\") pod \"must-gather-hbbcr\" (UID: \"92818b3a-57ba-43fa-823c-01d3bbdffef1\") " pod="openshift-must-gather-wn4x5/must-gather-hbbcr" Nov 03 15:02:53.986796 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:53.986654 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjd2w\" (UniqueName: \"kubernetes.io/projected/92818b3a-57ba-43fa-823c-01d3bbdffef1-kube-api-access-bjd2w\") pod \"must-gather-hbbcr\" (UID: \"92818b3a-57ba-43fa-823c-01d3bbdffef1\") " pod="openshift-must-gather-wn4x5/must-gather-hbbcr" Nov 03 15:02:54.088055 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:54.087975 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjd2w\" (UniqueName: \"kubernetes.io/projected/92818b3a-57ba-43fa-823c-01d3bbdffef1-kube-api-access-bjd2w\") pod \"must-gather-hbbcr\" (UID: \"92818b3a-57ba-43fa-823c-01d3bbdffef1\") " pod="openshift-must-gather-wn4x5/must-gather-hbbcr" Nov 03 15:02:54.088276 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:54.088059 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92818b3a-57ba-43fa-823c-01d3bbdffef1-must-gather-output\") pod \"must-gather-hbbcr\" (UID: \"92818b3a-57ba-43fa-823c-01d3bbdffef1\") " pod="openshift-must-gather-wn4x5/must-gather-hbbcr" Nov 03 15:02:54.088418 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:54.088397 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92818b3a-57ba-43fa-823c-01d3bbdffef1-must-gather-output\") pod \"must-gather-hbbcr\" (UID: \"92818b3a-57ba-43fa-823c-01d3bbdffef1\") " pod="openshift-must-gather-wn4x5/must-gather-hbbcr" Nov 03 15:02:54.145667 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:54.145635 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjd2w\" (UniqueName: \"kubernetes.io/projected/92818b3a-57ba-43fa-823c-01d3bbdffef1-kube-api-access-bjd2w\") pod \"must-gather-hbbcr\" (UID: \"92818b3a-57ba-43fa-823c-01d3bbdffef1\") " pod="openshift-must-gather-wn4x5/must-gather-hbbcr" Nov 03 15:02:54.180207 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:54.180180 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wn4x5/must-gather-hbbcr" Nov 03 15:02:54.546451 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:54.546425 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wn4x5/must-gather-hbbcr"] Nov 03 15:02:54.549892 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:54.549871 4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:02:54.669175 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:54.669140 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/must-gather-hbbcr" event={"ID":"92818b3a-57ba-43fa-823c-01d3bbdffef1","Type":"ContainerStarted","Data":"ca12fa3a87f716b626af4f095850a320262cf5b3bde4dd2da45d44b1a34ad7d6"} Nov 03 15:02:56.339449 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:56.339415 4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" limit=1024 Nov 03 15:02:56.683817 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:56.683782 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/must-gather-hbbcr" event={"ID":"92818b3a-57ba-43fa-823c-01d3bbdffef1","Type":"ContainerStarted","Data":"3ff680f7633b51ef0a8d435989d0a46fe699b54a89a11b9dfbe0d8036d190207"} Nov 03 15:02:56.684001 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:56.683827 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/must-gather-hbbcr" event={"ID":"92818b3a-57ba-43fa-823c-01d3bbdffef1","Type":"ContainerStarted","Data":"6f10dc9199a2239537d7ef4b6ded2d88ef13f9edef8d5e04fa74a2b37c95198e"} Nov 03 15:02:56.793392 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:02:56.793327 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wn4x5/must-gather-hbbcr" podStartSLOduration=2.397408544 podStartE2EDuration="3.793303512s" totalImagesPullingTime="1.395894968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:02:56.79104344 +0000 UTC m=+7109.985447034" watchObservedRunningTime="2026-11-03 15:02:56.793303512 +0000 UTC m=+7109.987707145" Nov 03 15:03:03.228618 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:03.228581 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6ct64_eed1b9f7-3769-46a0-b8ba-63eb8d17a952/global-pull-secret-syncer/0.log" Nov 03 15:03:04.334918 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:04.334890 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-92c7q_53c4e88e-1203-4083-98e9-9a2202a3a704/connection-checker/0.log" Nov 03 15:03:04.745094 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:04.744777 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-f6bgd_78248e2d-8c0b-47fb-b55b-07e0ffdc1c70/connection-checker/0.log" Nov 03 15:03:05.157895 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:05.156941 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-w6sm7_6365006c-9e6f-406f-9102-e3b0b452675d/connection-checker/0.log" Nov 03 15:03:05.627425 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:05.627341 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g6wbc_c6867ea1-4dd9-4ac3-8c5d-9ad6dd5dbcfc/konnectivity-agent/0.log" Nov 03 15:03:07.021838 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:07.021809 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-63.us-west-2.compute.internal_8df29524ca054c489efb3ae39305cb76/haproxy/0.log" Nov 03 15:03:07.861648 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:07.857011 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwnhg_19a01435-9815-42ed-9e2d-324a54f125bd/manager/0.log" Nov 03 15:03:08.127618 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:08.127510 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8kq74_dfd95a5f-d506-4461-8baf-44d26ec69181/manager/0.log" Nov 03 15:03:08.885833 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:08.885689 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xbk2m_bde28c41-ee98-42d6-ad61-6d08466d2fc9/manager/0.log" Nov 03 15:03:09.855047 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:09.855017 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-p9scb_d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb/server/0.log" Nov 03 15:03:11.314648 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:11.314606 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pghpt_3ff5388d-d285-446d-9ccc-98a7be123067/manager/0.log" Nov 03 15:03:12.923766 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:12.923735 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-qkwsd_1d3c8424-3ae1-488e-b37f-1d2a5d6c5199/manager/0.log" Nov 03 15:03:13.080543 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.080487 4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp"] Nov 03 15:03:13.087084 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.087028 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.139910 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.139868 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp"] Nov 03 15:03:13.241030 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.240943 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-podres\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.241030 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.240998 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-lib-modules\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.241030 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.241018 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88crr\" (UniqueName: \"kubernetes.io/projected/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-kube-api-access-88crr\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.241297 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.241143 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-proc\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.241297 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.241179 4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-sys\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.324454 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.324423 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-fvdmt_199f6b24-0d49-4637-b048-17b035f48627/manager/0.log" Nov 03 15:03:13.342261 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342220 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-sys\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.342458 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342339 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-podres\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.342458 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342394 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-lib-modules\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.342458 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342417 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88crr\" (UniqueName: \"kubernetes.io/projected/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-kube-api-access-88crr\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.342458 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342444 4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-proc\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.342881 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342579 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-proc\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.342881 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342638 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-sys\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.342881 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342691 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-podres\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.342881 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.342757 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-lib-modules\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.397993 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.397961 4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88crr\" (UniqueName: \"kubernetes.io/projected/60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c-kube-api-access-88crr\") pod \"perf-node-gather-daemonset-vs7tp\" (UID: \"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.402316 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.402284 4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:13.778285 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.778249 4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp"] Nov 03 15:03:13.780395 ip-10-0-129-63 kubenswrapper[4924]: W1103 15:03:13.780359 4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60e1f8fc_7c9e_4abd_a6c4_be6a88e4069c.slice/crio-72ba712286b7558d10c99de8c63d3a796607a4b3370e0c3498e29144470b6a96 WatchSource:0}: Error finding container 72ba712286b7558d10c99de8c63d3a796607a4b3370e0c3498e29144470b6a96: Status 404 returned error can't find the container with id 72ba712286b7558d10c99de8c63d3a796607a4b3370e0c3498e29144470b6a96 Nov 03 15:03:13.869234 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:13.869200 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" event={"ID":"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c","Type":"ContainerStarted","Data":"72ba712286b7558d10c99de8c63d3a796607a4b3370e0c3498e29144470b6a96"} Nov 03 15:03:14.153360 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:14.153284 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-x2vzl_a444f2ee-fc8a-4780-8058-3e5e6c2fae73/manager/0.log" Nov 03 15:03:14.655142 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:14.655104 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwnhg_19a01435-9815-42ed-9e2d-324a54f125bd/manager/0.log" Nov 03 15:03:14.878508 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:14.878456 4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" event={"ID":"60e1f8fc-7c9e-4abd-a6c4-be6a88e4069c","Type":"ContainerStarted","Data":"02b6bb6b247a170ba2bd8c30ee3afef2a337d4ec59900b8c0740b75d9564ca5b"} Nov 03 15:03:14.878737 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:14.878611 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:14.922894 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:14.922865 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8kq74_dfd95a5f-d506-4461-8baf-44d26ec69181/manager/0.log" Nov 03 15:03:14.989629 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:14.989565 4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" podStartSLOduration=1.989550032 podStartE2EDuration="1.989550032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:03:14.988383357 +0000 UTC m=+7128.182786952" watchObservedRunningTime="2026-11-03 15:03:14.989550032 +0000 UTC m=+7128.183953627" Nov 03 15:03:15.013719 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:15.013689 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mvjp9_7a0adcf2-59cc-4afa-b3ed-b403b1d1c67a/manager/0.log" Nov 03 15:03:15.554650 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:15.554615 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xbk2m_bde28c41-ee98-42d6-ad61-6d08466d2fc9/manager/0.log" Nov 03 15:03:16.400708 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:16.400682 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-p9scb_d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb/server/0.log" Nov 03 15:03:17.577519 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:17.577479 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pghpt_3ff5388d-d285-446d-9ccc-98a7be123067/manager/0.log" Nov 03 15:03:18.560360 ip-10-0-129-63 kubenswrapper[4924]: E1103 15:03:18.560313 4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cq7lf" containerName="extractor" Nov 03 15:03:20.473385 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:20.473347 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6cpnv_278ee3e6-54bd-4ae0-8ce1-fbda1caad4b2/cluster-monitoring-operator/0.log" Nov 03 15:03:20.729025 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:20.728949 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pwnhg_19a01435-9815-42ed-9e2d-324a54f125bd/manager/0.log" Nov 03 15:03:20.897128 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:20.897056 4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-vs7tp" Nov 03 15:03:20.991488 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:20.991414 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8kq74_dfd95a5f-d506-4461-8baf-44d26ec69181/manager/0.log" Nov 03 15:03:21.587967 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:21.587930 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xbk2m_bde28c41-ee98-42d6-ad61-6d08466d2fc9/manager/0.log" Nov 03 15:03:22.414676 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:22.414645 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-p9scb_d4e4cc55-7a0b-4bf8-bbad-355d9f45a9bb/server/0.log" Nov 03 15:03:23.594703 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:23.594666 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pghpt_3ff5388d-d285-446d-9ccc-98a7be123067/manager/0.log" Nov 03 15:03:24.300423 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:24.300390 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhgsp_aee646c2-b8d9-4998-b6e4-cffbfba7ea94/node-exporter/0.log" Nov 03 15:03:24.704559 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:24.704534 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhgsp_aee646c2-b8d9-4998-b6e4-cffbfba7ea94/kube-rbac-proxy/0.log" Nov 03 15:03:25.101942 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:25.101862 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vhgsp_aee646c2-b8d9-4998-b6e4-cffbfba7ea94/init-textfile/0.log" Nov 03 15:03:27.006387 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:27.006355 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fc46679f7cttvl_2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8/istio-proxy/0.log" Nov 03 15:03:29.170702 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:29.170661 4924 scope.go:122] "RemoveContainer" containerID="4998198d37e6df9b4d1267e8048d2a1a6b3e03fd659890e4536df81eebd56fde" Nov 03 15:03:29.772799 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:29.772770 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-dx6qt_43079ebb-3c46-4416-8822-bbe2be66119b/istio-proxy/0.log" Nov 03 15:03:30.852451 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:30.852416 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-nmndv_05cb7330-19b8-4112-b96b-cc70285298ed/istio-proxy/0.log" Nov 03 15:03:31.069939 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:31.069908 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-sw482_f586dcb0-04d2-4152-9556-931824aab29b/istio-proxy/0.log" Nov 03 15:03:31.285532 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:31.285506 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-59c5f4c899-ssp95_0db0b8cf-03c4-4493-a3f8-464723ef7eec/istio-proxy/0.log" Nov 03 15:03:31.943812 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:31.943782 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8478467d49-l5dtm_f11da27b-1a3d-4fca-b78e-fe9cd3caa46b/router/0.log" Nov 03 15:03:36.125940 ip-10-0-129-63 kubenswrapper[4924]: I1103 15:03:36.125914 4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fc46679f7cttvl_2ff10ee8-0bd0-465b-ba31-cf54dbcbd6e8/istio-proxy/0.log"